feat: alerts recovery threshold

[abhishekhugetech]

Added recovery threshold handling in prom, threshold and anomaly rules

Caution

Need to think on rollout strategy, either to ship it in one go or behind a feature flag.

📄 Summary

---

✅ Changes

• Feature: Brief description
• Bug fix: Brief description

---

🏷️ Required: Add Relevant Labels

⚠️ Manually add appropriate labels in the PR sidebar
Please select one or more labels (as applicable):

ex:

• frontend
• backend
• devops
• bug
• enhancement
• ui
• test

---

👥 Reviewers

Tag the relevant teams for review:

• frontend / backend / devops

---

🧪 How to Test

1. Generate data series in the following format as below.
2. Create alert with following configuration.
3. Start sending data

Generating data series

phase_duration_minutes = 3
# generates test data for recovery threshold alert
# Alert set at 10 with recovery at 7
def recovery_threshold_alert_data(elapsed_minutes):
    # normal data in start and end of test
    if elapsed_minutes < 1 or elapsed_minutes > 20:
        return 6
    # in each of the 2nd iteration we have flapping data
    if int((elapsed_minutes / phase_duration_minutes) % 2) == 1:
        return 20
    # in each of the 1st iteration we have data just below alert
    return 9

Alert config

Recovery threshold: 7
Alert condition:

Test Report

---

🔍 Related Issues

Closes #

---

📸 Screenshots / Screen Recording (if applicable / mandatory for UI related changes)

---

📋 Checklist

• Dev Review
• Test cases added (Unit/ Integration / E2E)
• Manually tested the changes

---

👀 Notes for Reviewers

[welcome]

Welcome to the SigNoz community! Thank you for your first pull request and making this project better. 🤗

[CLAassistant]

All committers have signed the CLA.

[srikanthccv]

Need to think on rollout strategy, either to ship it in one go or behind a feature flag.

Is there any reason to put it behind a feature flag?