Skip to content

Document how to debug performance of Prow jobs in etcd #20796

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Document how to debug performance of Prow jobs in etcd #20796

wants to merge 1 commit into from
+108 −0

Conversation

@ronaldngounou
Copy link
Member

Overview

This PR documents how etcd contributors can navigate performance of prow jobs.
Contributes to issue #18707
@k8s-ci-robot
Copy link

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all
@k8s-ci-robot
Copy link

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: ronaldngounou
Once this PR has been reviewed and has the lgtm label, please assign ivanvc for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment
@ronaldngounou ronaldngounou force-pushed the debug-performance-prow branch from eeca3a0 to 2e38276 Compare October 15, 2025 07:59
@ronaldngounou ronaldngounou mentioned this pull request Oct 16, 2025
Open
@ronaldngounou ronaldngounou force-pushed the debug-performance-prow branch from 2e38276 to 6d3c4f2 Compare October 19, 2025 06:49
@ronaldngounou ronaldngounou marked this pull request as ready for review October 19, 2025 06:52
@ronaldngounou ronaldngounou force-pushed the debug-performance-prow branch 5 times, most recently from 822a203 to eb48c66 Compare October 19, 2025 07:22
@codecov
Copy link

codecov bot commented Oct 19, 2025 •
edited
Loading

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 69.18%. Comparing base (8a4955b) to head (e6a773b).

Additional details and impacted files

see 21 files with indirect coverage changes

@@            Coverage Diff             @@
##             main   #20796      +/-   ##
==========================================
+ Coverage   69.12%   69.18%   +0.06%     
==========================================
  Files         422      422              
  Lines       34826    34826              
==========================================
+ Hits        24073    24095      +22     
+ Misses       9352     9333      -19     
+ Partials     1401     1398       -3

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8a4955b...e6a773b. Read the comment docs.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
@ronaldngounou
Copy link
Member Author

ronaldngounou commented Oct 22, 2025 •
edited
Loading

Hi @ivanvc , could you please review?
Is there anything I could do to resolve the failing tests ?
@serathius
Copy link
Member

Is there anything I could do to resolve thr failing tests ?

Please rebase
@ivanvc
Copy link
Member

ivanvc commented Oct 23, 2025

Hi @ronaldngounou, there are a couple of broken links in the document you pushed. Refer to the Prow job failure: https://prow.k8s.io/view/gs/kubernetes-ci-logs/pr-logs/pull/etcd-io_etcd/20796/pull-etcd-verify/1979810317445632000#

Found broken reference (Prow -> ) at /home/prow/go/src/github.com/etcd-io/etcd/Documentation/contributor-guide/prow_jobs.md:5
Found broken path (dashboards -> monitoring-gke.prow.k8s.io/d/96Q8oOOZk/builds?orgId=1&refresh=30s&var-org=etcd-io&var-repo=etcd&var-build=All&from=now-7d&to=now) at /home/prow/go/src/github.com/etcd-io/etcd/Documentation/contributor-guide/prow_jobs.md:41
@ronaldngounou ronaldngounou force-pushed the debug-performance-prow branch from eb48c66 to 5598b5d Compare October 25, 2025 02:58
@ronaldngounou
Copy link
Member Author

I have rebased and fixed the broken links.
cc @ivanvc @serathius
ivanvc
ivanvc reviewed Oct 26, 2025
View reviewed changes
Copy link
Member

@ivanvc ivanvc left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, @ronaldngounou. This is a great start. I left some comments. Please take a look at them.
@ronaldngounou
Copy link
Member Author

I will address the feedback as soon as possible. I am primary oncall this week
@jberkus
Copy link

jberkus commented Oct 29, 2025

/assign
@ronaldngounou ronaldngounou force-pushed the debug-performance-prow branch 3 times, most recently from 261530a to 56f9a51 Compare October 31, 2025 07:40
@ronaldngounou
Copy link
Member Author

/retest
ivanvc
ivanvc reviewed Oct 31, 2025
View reviewed changes
Copy link
Member

@ivanvc ivanvc left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the updates, @ronaldngounou. I left some comments after the update.
Documentation/contributor-guide/prow_jobs.md
Comment on lines 88 to 93
## 3.1 Prow job categories (robustness, integration, static checks)

- Robustness:
- Description: Long-running, fault-injection and chaos-style end-to-end tests that validate etcd correctness and availability under failures (node crashes, network partitions, resource exhaustion, upgrades).
- When to run: Periodics for continuous coverage; run for PRs that touch consensus, storage, recovery, or upgrade paths.
- Example job patterns: pull-etcd-robustness, periodic-robustness

- Integration:
- Description: Functional end-to-end and cross-component tests that exercise real client/server interactions, snapshots/restore, upgrades and compatibility across OS/arch.
- When to run: Presubmits for PRs that change APIs, client behavior, or integration points; periodics for broad platform coverage.
- Example job patterns: pull-etcd-e2e-amd64, pull-etcd-integration

- Static check:
- Description: Fast, deterministic checks (build, unit tests, linters, go vet/staticcheck, formatting, license/header checks, generated-code verification) that catch style, correctness and packaging problems early.
- When to run: Every PR as presubmits; quick feedback loop before running expensive tests.
- Example job patterns: pull-etcd-verify, pull-etcd-lint, pull-etcd-unit
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure if this classification is right. I wouldn't consider unit tests static checks. Maybe a better classification is Static Checks and Tests. Tests itself is break down into Robustness, Integration, E2E, Unit, Release (and maybe others?). Static checks are linters, vulnerability analysis, etc.
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Alright. I have reordered the structure. Let me know what you think and I can adjust the restructure.
@ronaldngounou
Document how Prow jobs are used in etcd
e6a773b 
- Introduction to Prow
- How Prow is used for etcd testing
- Navigating perfomance dashboards (Grafana)
- Prow job categories
- Interpreting metrics

Signed-off-by: ronaldngounou <ronald.ngounou@yahoo.com>
@ronaldngounou ronaldngounou force-pushed the debug-performance-prow branch from 56f9a51 to e6a773b Compare November 1, 2025 01:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/documentation size/L

5 participants

@ronaldngounou @k8s-ci-robot @serathius @ivanvc @jberkus