Description
For now metrics are only associated with Pod
resources:
elasticsearch-k8s-metrics-adapter/pkg/client/elasticsearch/discovery.go
Lines 195 to 200 in 39a36d9
Also namespace and Pod's name fields are hardcoded in the Elasticsearch query:
{
"query": {
"bool": {
"must": [{
"exists": {
"field": "%s"
}
}, {
"match": {
"kubernetes.namespace": "%s" -- here
}
}, {
"match": {
"kubernetes.pod.name": "%s" -- here
}
}]
}
}
}
Not all metrics make sense in the context of Pods. For example it does not make sense to expose system.memory.free
as it is not possible to associate this metric to a specific Pod. The first approach so far has been to filter on the metric name and assume documents also have the kubernetes.namespace
and kubernetes.pod.name
fields:
metricSets:
- indices: [ 'metricbeat-*' ]
fields:
- patterns: [ '^prometheus\.metrics\.' , '^kibana\.stats\.' ] # only serve metrics with namespace and pod name
We then assume that we get that kind of document from Elasticsearch:
{
"_index": "metricbeat-7.14.0-2021.09.21-000001",
"_type": "_doc",
"_id": "10V6DXwBJBC0gYXF1JYC",
"fields": {
"prometheus.metrics.go_memstats_frees_total": [
123418991
],
[ ... other metrics starting with prometheus.metrics* ...]
"kubernetes.pod.name": [
"kube-dns-b4f5c58c7-47twh"
],
"kubernetes.namespace": [
"kube-system"
],
"@timestamp": [
"2021-09-22T12:28:44.448Z"
]
}
}
We may want to improve this association mechanism, for example by allowing other resources to be associated with a given metric. Users may also want to use other fields than kubernetes.namespace
or kubernetes.pod.name
to query for a given metric.
See here how associations are managed with the Prometheus adapter.