feat: write open telemetry semantic conventions for AI metrics and span attributes #1601
Conversation
| }; | ||
| return timedResponse; | ||
| }); | ||
| return runner(input, getStreamingCallback(registry)).then( |
There was a problem hiding this comment.
Should we just use the await syntax here, since we're introducing it below (await writeSemConv...) so that we're consistent?
| export interface SemConvOptions { | ||
| writeMetrics: boolean; | ||
| writeSpanAttributes: boolean; | ||
| } |
There was a problem hiding this comment.
What's the use case for turning this on and off at this level?
I am wondering if we should always write metrics, and then let the client/exporter control whether or not the metric actually gets sent anywhere. I think this is how could work? But correct me if I am off base.
There was a problem hiding this comment.
Couple other thoughts...
- Semantic Convention is a generic term, there are many conventions for many things (e.g. HTTP) so if this is a flag we want, it should be specific as to what convention it refers to.
- I imagine we should have a generic way of allow-listing, deny-listing metrics that are exported, versus a metric specific flag.
| system: 'vertex_ai', | ||
| requestModel: request.config?.version || model.version || name, | ||
| responseModel: response['modelVersion'], | ||
| operationName: 'image_generation', |
There was a problem hiding this comment.
should this be the actual endpoint being called? if so, I think its predict
| system: 'google_ai', | ||
| requestModel: modelName, | ||
| responseModel: response['modelVersion'], | ||
| operationName: 'chat', |
There was a problem hiding this comment.
Is startChat in the SDK a veneer on top of generateContent w/ session management? I am iffy what the operationName should be.
- Genkit Operation
- Gemini SDK operation
- Underlying Gemini API operation
|
Overall this looks good, however gen ai semantic conventions are experimental. How do you plan to manage the potentially breaking changes in the convention? |
4 participants
Write Metrics and Span attributes matching the Semantic Conventions for Generative AI systems
Checklist (if applicable):