Skip to content

io: MultiReader should be more efficient when chained multiReaders contain few actual readers #13558

@ajwerner

Description

@ajwerner

The specific motivating case here is when doing decoding from a stream of json. In some cases I need to make two passes to determine the appropriate struct into which to decode based on some content. The json.Decoder offers buffered in an io.Reader. Thus I have access to the in-tact stream by taking:

io.MultiReader(dec.Buffered(), r) // where r is the underlying reader.

The problem with this is that each time this process happens, a new multiReader is created to wrap these two readers. In general, the next call to read will Read the entire Reader from Buffered leaving just the underlying reader. The problem is that the last reader remains wrapped in an underlying *multiReader. So after doing this 100 times, there may be just one actual reader with data but the call to read will have to call through the multiReader.Read method 100 times.

The current constructor looks like this:

func MultiReader(readers ...Reader) Reader {
    r := make([]Reader, len(readers))
    copy(r, readers)
    return &multiReader{r}
}

I propose that we do something like this:

func MultiReader(readers ...Reader) Reader {
    r := make([]Reader, 0, len(readers))
    for _, reader := range readers {
        if mr, ok := reader.(*multiReader); ok {
            r = append(r, mr.readers...)
        } else {
            r = append(r, reader)
        }
    } 
    return &multiReader{r}
}

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions