Skip to content

Change the log-level when a new ServerTransport cannot be created #5523

@jpkrohling

Description

@jpkrohling

Use case(s) - what problem will this feature solve?

Both on the Jaeger and on the OpenTelemetry communities, we've seen users complaining about the logs being spammed by messages like "grpc: Server.Serve failed to create ServerTransport" being recorded periodically. This is typically caused by load balancers or by liveness probes. As such, they don't deserve to be set at the warning level.

I would like then to suggest the following code be changed to Info-level warning instead of Warning. The log-levels document would likely need to be changed or refined to state that new connections might fail, whereas failure on established connections might still be logged at warning levels.

grpc-go/server.go

Lines 900 to 902 in f601dfa

if err != io.EOF {
channelz.Warning(logger, s.channelzID, "grpc: Server.Serve failed to create ServerTransport: ", err)
}

Proposed Solution

Change the log-level for failure to establish new connections to Info.

Alternatives Considered

  1. I considered wrapping the logger on my component which would suppress this particular message, but that sounds just wrong.
  2. Create a new component for this (and similar events) here, with the appropriate log-level. That doesn't seem very clean either.

Additional Context

open-telemetry/opentelemetry-collector#4747
#4234

This can be reproduced with OpenTelemetry Collector, with the OTLP receiver enabled and running a cURL call like: curl localhost:4317. It will yield the following:

2022-07-19T16:34:23.752-0300	warn	zapgrpc/zapgrpc.go:191	[core] [Server #1] grpc: Server.Serve failed to create ServerTransport: connection error: desc = "transport: http2Server.HandleStreams received bogus greeting from client: \"GET / HTTP/1.1\\r\\nHost: lo\""	{"grpc_log": true}

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type: FeatureNew features or improvements in behavior

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions