Mattermost Integration: Chat message does not update when alert group changes #5568
Description
What went wrong?
What happened:
We started using the Mattermost integration and encountered an issue: when changes occur in an alert group (for example, a new alert is added to the alert group), the message in Mattermost does not update and the changes are not reflected. Previously, we used the Telegram integration, and any changes in the alert group were immediately reflected in the OnCall message.
What did you expect to happen:
I expected the message in Mattermost to update automatically when the alert group changes, and new alerts to be displayed in the OnCall message.
How do we reproduce it?
- Make sure you have Mattermost integration configured.
- Create an alert, for example, that node-exporter is down on node meow-01 (please replace INTEGRATION_WEBHOOK):
curl --header "Content-Type: application/json" --data '{"receiver":"grafana-oncall","status":"firing","alerts":[{"status":"firing","labels":{"alertname":"targetDown","instance":"meow-01","job":"node-exporter","severity":"critical"},"annotations":{"message":"node-exporter is down on meow-01"},"startsAt":"2025-08-18T05:25:15Z","endsAt":"0001-01-01T00:00:00Z","generatorURL":"http://meowmeow","fingerprint":"meowmeowmeowmeow01"}],"groupLabels":{"alertname":"targetDown"},"commonLabels":{"alertname":"targetDown","severity":"critical"},"commonAnnotations":{"message":"node-exporter is down on meow-01"},"externalURL":"http://meowmeowmeow:9093","version":"4","groupKey":"{}/{severity=~\"(critical)\"}:{alertname=\"targetDown\"}","truncatedAlerts":0,"numFiring":1,"numResolved":0}' https://INTEGRATION_WEBHOOK
- Make sure that a message about the alert appears in Mattermost.
- Add a change to the alert group, for example, that node-exporter has become unavailable on another node, meow-02:
curl --header "Content-Type: application/json" --data '{"receiver":"grafana-oncall","status":"firing","alerts":[{"status":"firing","labels":{"alertname":"targetDown","instance":"meow-01","job":"node-exporter","severity":"critical"},"annotations":{"message":"node-exporter is down on meow-01"},"startsAt":"2025-08-18T05:25:15Z","endsAt":"0001-01-01T00:00:00Z","generatorURL":"http://meowmeow","fingerprint":"meowmeowmeowmeow01"},{"status":"firing","labels":{"alertname":"targetDown","instance":"meow-02","job":"node-exporter","severity":"critical"},"annotations":{"message":"node-exporter is down on meow-02"},"startsAt":"2025-08-18T05:37:00Z","endsAt":"0001-01-01T00:00:00Z","generatorURL":"http://meowmeow","fingerprint":"meowmeowmeowmeow02"}],"groupLabels":{"alertname":"targetDown"},"commonLabels":{"alertname":"targetDown","severity":"critical"},"commonAnnotations":{},"externalURL":"http://meowmeowmeow:9093","version":"4","groupKey":"{}/{severity=~\"(critical)\"}:{alertname=\"targetDown\"}","truncatedAlerts":0,"numFiring":2,"numResolved":0}' https://INTEGRATION_WEBHOOK
- Check that the message in Mattermost has not been updated and does not contain information about the new alert on meow-02.
Grafana OnCall Version
v1.16.4
Product Area
Chatops
Grafana OnCall Platform?
Docker
User's Browser?
No response
Anything else to add?
It is worth noting that for the events "Acknowledged" and "Resolved", the messages in Mattermost are updated correctly. In addition, we see the following error in the celery logs every time changes occur in the alert group:
2025-08-18 06:02:01,982 source=engine:celery worker=ForkPoolWorker-2 task_id=06373ff6-617e-4b25-af61-57a497e8b5c7 task_name=apps.mattermost.tasks.on_create_alert_async name=apps.mattermost.tasks level=ERROR Mattermost message exist with post id 13tpmtiiwiyx5esi1qrh4agk4c hence skipping
Our Mattermost version is v10.5.7.