Skip to content

reuploader: walk only S3_PREFIX and submit only .post objects - #1310

Open
aagbsn wants to merge 1 commit into
masterfrom
reuploader_s3_prefix
Open

aagbsn wants to merge 1 commit into
masterfrom
reuploader_s3_prefix

Conversation

@aagbsn

@aagbsn aagbsn commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

use S3_PREFIX so that the failed-reports bucket can hold a dedup key (alongside postcans), for reprocessing of measurements that got doubly processed by resubmitting into fastpath. fixes the previously ignored S3_PREFIX and adds a test to confirm it behaves as expected. Skip keys that don't end in .post.
Fix the variable names in the Dockerfile and README (S3_BUCKET_NAME, S3_PREFIX), and the Dockerfile's AWS_REGION line, whose spaces around "=" made it part of the value.

main() read S3_PREFIX but walked the bucket from the top, so the
reuploader sent every object in the failed-reports bucket and deleted
each one fastpath accepted. Fastpath accepts any body, so that included
objects that aren't measurements.

Walk from S3_PREFIX and skip keys that don't end in .post. Fix the
variable names in the Dockerfile and README (S3_BUCKET_NAME, S3_PREFIX),
and the Dockerfile's AWS_REGION line, whose spaces around "=" made it
part of the value. Add tests and a workflow to run them.
Comment thread reuploader/Makefile
echo '[{"name":"${ECS_CONTAINER_NAME}","imageUri":"${IMAGE_NAME}:${BUILD_LABEL}"}]' > imagedefinitions.json

test:
pip install -q . pytest

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why not implementing the test command like other commands in backend? With hatch, see

test:
hatch run test

@LDiazN LDiazN Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And also you're potentially installing pytest globally instead of inside a virtualenv, hatch helps with this.

It would also be nice to add test as a .PHONY target

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

2 participants