Simple pipe reader for hdfs or other service

[typhoonzero]

Fix #5011

[helinwang]

Maybe we just need to support "plain", the user can decompress it outside of Paddle using pipe.

[typhoonzero]

Thought it may be inconvenient for users to decompress stream data in their parsers.

[helinwang]

I meant the user can decompress the data using shell commands, not in the parsers, e.g.:

hadoop fs -cat /path/to/some/file | gzip -d

[typhoonzero]

Well, this is simpler, but I'm considering the pipe size using bash is set by ulimit, when in cluster trainer, users may not have control over every node's ulimit configuration, but using python code can.

[helinwang]

I don't understand bash very well, but does the pipe just "block" if it's full, and probably gzip can decode in a stream fashion, and will consume the pipe buffer, so it will be unblocked.

[typhoonzero]

By default, pipes can block both producer and consumer:

If a process attempts to read from an empty pipe, then read(2) will
block until data is available. If a process attempts to write to a
full pipe (see below), then write(2) blocks until sufficient data has
been read from the pipe to allow the write to complete.

Well, my point is, use pipes in python code, can let users to define pipe buffer size which is critical to the reader performance.

[helinwang]

I see, ok. Thanks!

[helinwang]

line break won't work in binary data, maybe we should let parser decide when to output a new data item?

[typhoonzero]

If cut_lines=False the binary data will send to parser directly. Do you mean by should let user's parser generate data, and make pipe_reader a decorator?

[helinwang]

Yes, I thought maybe pipe_reader should not cut the lines, since it does not have sufficient information, we might want leave it to the user's parser to do so (cut and generate data).

[typhoonzero]

Agree, will update.

[helinwang]

LGTM! Let's update https://github.com/PaddlePaddle/Paddle/pull/5282/files#r150348324 with a follow up commit.