Python Popen().stdout.read() hang

Question

I'm trying to get output of another script, using Python's subprocess.Popen like follows

process = Popen(command, stdout=PIPE, shell=True)
exitcode = process.wait()
output = process.stdout.read()   # hangs here

It hangs at the third line, only when I run it as a python script and I cannot reproduce this in the python shell.

The other script prints just a few words and I am assuming that it's not a buffer issue.

Does anyone has idea about what I am doing wrong here?

And if you run the command in command manually, it generates output? — Some programmer dude
– Some programmer dude, Commented Jan 21, 2013 at 6:59

Amber · Accepted Answer · 2013-01-21 06:59:37Z

5

You probably want to use .communicate() rather than .wait() plus .read(). Note the warning about wait() on the subprocess documentation page:

Warning This will deadlock when using stdout=PIPE and/or stderr=PIPE and the child process generates enough output to a pipe such that it blocks waiting for the OS pipe buffer to accept more data. Use communicate() to avoid that.

http://docs.python.org/2/library/subprocess.html#subprocess.Popen.wait

answered Jan 21, 2013 at 6:59

Amber

531k89 gold badges643 silver badges558 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

lyomi Over a year ago

Thanks for the answer. I just realized I have read that part of the docs.. What I was doing was fetching the stdout real-time in a while process.poll() is None: loop while the script is running, and the code above was a stripped down version. But I guess it was hanging because of a similar deadlock issue ...

crxyz Over a year ago

hmm this didn't work for me, it still hangs

aaa90210 · Accepted Answer · 2017-11-14 23:54:02Z

1

read() waits for EOF before returning.

You can:

wait for the subprocess to die, then read() will return.
use readline() if your output is broken into lines (will still hang if no output lines).
use os.read(F,N) which returns at most N bytes from F, but will still block if the pipe is empty (unless O_NONBLOCK is set on the fd).

answered Nov 14, 2017 at 23:54

aaa90210

12.3k14 gold badges56 silver badges93 bronze badges

1 Comment

user518450 Over a year ago

You can use unbuffered file objects for your pipes if you put bufsize=0 in the popen call. This helps of you have the problem of the buffer not filling up. For bufsize=0 the read may come back with less bytes then requested before it blocks on the empty file object.

sergzach · Accepted Answer · 2019-03-23 16:55:44Z

0

You can see how to deal with hanging reading of stdout/stderr in the next sources:

readingproc

edited Mar 23, 2019 at 16:55

answered Mar 23, 2019 at 16:20

sergzach

6,8047 gold badges52 silver badges88 bronze badges

Collectives™ on Stack Overflow

Python Popen().stdout.read() hang

3 Answers 3

2 Comments

1 Comment

Comments

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

2 Comments

1 Comment

Comments

Linked

Related