Skip to content

Add end to end tests - #160

Open
aagbsn wants to merge 79 commits into
mainfrom
add_end_to_end_tests
Open

Add end to end tests#160
aagbsn wants to merge 79 commits into
mainfrom
add_end_to_end_tests

Conversation

@aagbsn

@aagbsn aagbsn commented Mar 24, 2026

Copy link
Copy Markdown
Contributor

No description provided.

aagbsn added 30 commits March 11, 2026 13:35
@aagbsn
add end-to-end tests with docker-compose.yml
d1b3566
@aagbsn
run workflow on this branch
19870b3
@aagbsn
build oonipipeline as command
66366ec
@aagbsn
fix typo
303b276
@aagbsn
add missing quote
97629dc
@aagbsn
remove deprecated API endpoint
543856d
@aagbsn
add clickhouse_init.sql
7d8120a
@aagbsn
run fastpath_feeder
3e36643
@aagbsn
integ: postgres, valkey, and api-oonimeasurements
52b5db3 
add these services to docker-compose.yml
@aagbsn
fix VALKEY_URL
f2f0b96
@aagbsn
use healthcheck and depends_on: condition to stage services
3224977
@aagbsn
use CLICKHOUSE_URL in environment
7609b13
@aagbsn
add healthcheck to api
faaf2d8
@aagbsn
set higher rate limits
c14ba81
@aagbsn
use depends_on: condition: to stage starting oonimeasurements
30b675e
@aagbsn
add verify to run oonimeasurements tests against api
7f50e11
@aagbsn
add api tests
a2aec7b
@aagbsn
add fastpath table to init
077e234
@aagbsn
use date range of downloaded data
05c0dfc 
FIXME: parametize this
@aagbsn
use right engine
4a60f1a
@aagbsn
fix date range in conftest.py
dd04a63
@aagbsn
Merge remote-tracking branch 'origin/main' into add_end_to_end_tests
5a32e99
@aagbsn
wait until fastpath has completed successfully
ce707a6
@aagbsn
extend date range
b61ebbd
@aagbsn
add /var/lib/fastpath to fastpath volumes
af70027
@aagbsn
fix since and until date range
9b980a9 
diff --git a/tests/integration/tests/conftest.py b/tests/integration/tests/conftest.py
index 36b7ca6..8ca007e 100644
--- a/tests/integration/tests/conftest.py
+++ b/tests/integration/tests/conftest.py
@@ -98,7 +98,7 @@ def params_since_and_until_with_two_days():

 @pytest.fixture
 def params_since_and_until_with_ten_days():
-    return set_since_and_until_params(since="2026-01-01", until="2026-01-02")
+    return set_since_and_until_params(since="2026-01-01", until="2026-01-10")

 def set_since_and_until_params(since, until):
diff --git a/tests/integration/tests/test_oonidata_aggregate_analysis.py b/tests/integration/tests/test_oonidata_aggregate_analysis.py
index 23762bd..6bad311 100644
--- a/tests/integration/tests/test_oonidata_aggregate_analysis.py
+++ b/tests/integration/tests/test_oonidata_aggregate_analysis.py
@@ -3,7 +3,7 @@ import os

 route = "/api/v1/aggregation/analysis"
 since = "2026-01-01"
-until = "2026-01-02"
+until = "2026-01-10"

 def test_oonidata_aggregation_analysis(client):
diff --git a/tests/integration/tests/test_oonidata_list_analysis.py b/tests/integration/tests/test_oonidata_list_analysis.py
index 1be6fce..64af92e 100644
--- a/tests/integration/tests/test_oonidata_list_analysis.py
+++ b/tests/integration/tests/test_oonidata_list_analysis.py
@@ -2,7 +2,7 @@ import pytest

 route = "/api/v1/analysis"
 since = "2026-01-01"
-until = "2026-01-02"
+until = "2026-01-10"

 def test_oonidata_list_analysis(client):
@aagbsn
set start-day and end-day rnage in run_fastpath
ebe4334 
--- a/tests/integration/docker-compose.yml
+++ b/tests/integration/docker-compose.yml
@@ -57,7 +57,7 @@ services:
     environment:
       CLICKHOUSE_URL: "http://testuser:testuser@clickhouse:9000/ooni"
     command: >
-      bash -c "./run_fastpath --debug --clickhouse-url $CLICKHOUSE_URL  --stdout"
+      bash -c "./run_fastpath --debug --clickhouse-url $CLICKHOUSE_URL --keep-s3-cache --write-to-disk --stdout --start-day 2026-01-01 --end-day 2026-01-10 --noapi"
     depends_on:
       clickhouse:
         condition: service_healthy
@aagbsn
update tests to use reports in dataset
e4bd778
@aagbsn
add fastpath.conf
be374ce
@aagbsn
fastpath: filter --ccs IR,IT
40838c7
@aagbsn
aagbsn force-pushed the add_end_to_end_tests branch from 2ef34e1 to 6d94632 Compare April 20, 2026 11:27
@aagbsn
aagbsn force-pushed the add_end_to_end_tests branch from ba5f89a to 2abfea4 Compare April 20, 2026 13:55
@aagbsn aagbsn mentioned this pull request Apr 21, 2026
Open
aagbsn added a commit to ooni/devops that referenced this pull request Aug 18, 2026
@aagbsn
Add real-data end-to-end upgrade scenario (ooni/data#160)
a820098 
Extends the harness with a second, additional CI job that verifies the
actual OONI data pipeline -- not just synthetic seed data -- survives the
production upgrade path. Design per direct instruction: download real
data exactly once per run, re-verify integrity + query correctness at
each hop, and walk PRODUCTION_HOPS (the real 4-hop runbook) rather than
every diagnostic bisection waypoint.

- harness/real_data.py: new module. Snapshots real-data tables (row count
  + order-independent cityHash64 checksum) across all 3 nodes, loads real
  OONI measurements once via ooni/data#160's downloader/fastpath
  containers, takes a golden snapshot, then per hop: upgrades all 3 nodes
  (reusing scenarios.upgrade_node_step unmodified), diffs against the
  golden snapshot, and re-runs ooni/data's own pytest suite against the
  real api-oonimeasurements service.
- docker-compose.real-data.yml: new overlay adapting ooni/data#160's
  tests/integration stack onto this project's existing 3-node replicated
  cluster instead of a single throwaway node. Pinned to the
  add_end_to_end_tests branch since #160 is still unmerged -- update the
  ref once it merges.
- sql/001_schema.sql: adds fingerprints_dns, fingerprints_http,
  obs_web_ctrl, obs_http_middlebox, obs_openvpn -- the tables
  oonipipeline's observations workflow and fastpath actually write, with
  per-table provenance notes and manual signedness fixes where
  oonipipeline's own DDL generator is known-wrong (Int32/Int8 vs the
  already-verified obs_web table's UInt32/UInt8 convention).
- harness/compose.py: multi-compose-file support (files= param throughout)
  plus inspect_exit_code()/run_oneoff() for polling one-shot containers
  and running a fresh `verify` pytest pass per hop.
- harness/scenarios.py: extracted apply_schema() from
  load_schema_and_seed() so the real-data scenario can reuse the schema
  load without the synthetic seed rows; generalized step_ok() to cover
  every non-upgrade-node step shape via a single top-level "ok" fallback.
- harness/report.py: render_ci_step() now handles the real-data scenario's
  step shapes (setup-real-data, load-real-data, golden-snapshot,
  verify-e2e, real-data-hop).
- ci_step.py: 6 new subcommands (setup-real-data, load-real-data,
  golden-snapshot, verify-e2e, real-data-hop, teardown-real-data) wiring
  harness/real_data.py into discrete, individually pass/fail-able CI
  steps, matching the existing setup/upgrade-node/verify-ddl pattern.
- .github/workflows/clickhouse_upgrade_test.yml: new real-data-upgrade
  job (checkout ooni/data at add_end_to_end_tests, sanity-check
  PRODUCTION_HOPS alignment, setup/load/snapshot/4-hop-loop/report/
  teardown). Deliberately not run on pull_request -- depends on an
  unmerged external branch and real network downloads -- only on push to
  main and workflow_dispatch (scenario: real-data or all).
- README.md: documents the new job's design, the load-once/verify-per-hop
  rationale, the external-branch caveat, and marks PR #477 review point 4
  ("run the target version against the real API + data pipeline") as
  addressed.

Verified offline (no Docker daemon available in this sandbox, consistent
with every prior patch in this project): all Python modules compile, all
YAML/compose files parse and interpolate, SQL is structurally sound
(balanced parens/backticks, unique table names), and every new step shape
round-trips correctly through step_ok()/render_ci_step(). Real validation
deferred to an actual GitHub Actions run.
hellais
hellais requested changes Aug 18, 2026
View reviewed changes

@hellais hellais left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Once the changes to the API land we can merge this
Comment thread tests/integration/tests/conftest.py Resolved
@aagbsn

aagbsn commented Sep 21, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

edit: command line arguments changed; fixed now
@aagbsn aagbsn mentioned this pull request Sep 27, 2026
Draft
@aagbsn
aagbsn requested a review from hellais September 28, 2026 07:35
hellais
hellais reviewed Sep 28, 2026
View reviewed changes
Comment thread tests/integration/clickhouse_init.sql Outdated
)
ENGINE = ReplacingMergeTree(ts)
ORDER BY (probe_asn, probe_cc, domain)
SETTINGS index_granularity = 8192;

@hellais hellais Sep 28, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The schemas for these tables are already defined inside of ooni/data itself and can be bootstrapped using the checkdb command https://github.com/ooni/data/blob/main/oonipipeline/src/oonipipeline/cli/commands.py#L238 or manually calling the create tables function like this: https://github.com/ooni/data/blob/main/oonipipeline/src/oonipipeline/db/create_tables.py#L492

@aagbsn aagbsn Oct 8, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

added
hellais
hellais reviewed Sep 28, 2026
View reviewed changes
Comment thread tests/integration/tests/test_oonidata_aggregate_analysis.py

json = response.json()
assert isinstance(json["results"], list), json
assert len(json["results"]) == total

@hellais hellais Sep 28, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we do something in some of these tests to validate more than just the schema of the returned formats? I think even just checking if some value is in there would go a long way.
hellais
hellais requested changes Sep 28, 2026
View reviewed changes

@hellais hellais left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is looking good. I would suggest pulling in the schema from oonipipeline so we aren't duplicating it and if possible add some additional checks for the values in the API responses.
aagbsn added 3 commits October 8, 2026 09:01
@aagbsn
Merge remote-tracking branch 'origin/main' into add_end_to_end_tests
acb4092
@aagbsn
e2e: create the pipeline tables with oonipipeline checkdb
36f5c59 
clickhouse_init.sql copied the fingerprints and event_detector_cusums
tables from oonipipeline. A create_tables service now creates every
pipeline table with oonipipeline checkdb --create-tables before
fastpath and the downloader start, since fastpath reads the
fingerprints tables. clickhouse_init.sql keeps only fastpath's own
tables, fastpath and obs_openvpn, which ooni/data doesn't define.
@aagbsn
e2e: check values in the API responses, not only their shape
a117cd6 
The pipeline loads IT and IR measurements from 2026-01-01 to 01-03, so
results must come from those countries and fall in the requested
window; aggregation counts must be positive, outcome likelihoods
between 0 and 1, and observation totals the same whichever way they are
grouped.
@aagbsn
aagbsn requested a review from hellais October 8, 2026 11:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

2 participants

@aagbsn @hellais