Parallelize the unit and integration tests#520
Draft
kmontemayor2-sc wants to merge 13 commits intomainfrom
Draft
Parallelize the unit and integration tests#520kmontemayor2-sc wants to merge 13 commits intomainfrom
kmontemayor2-sc wants to merge 13 commits intomainfrom
Conversation
Collaborator
Author
|
/unit_test_py |
Collaborator
Author
|
/integration_test |
Contributor
GiGL Automation@ 03:39:24UTC : 🔄 @ 04:55:30UTC : ✅ Workflow completed successfully. |
Contributor
GiGL Automation@ 03:39:30UTC : 🔄 @ 05:02:55UTC : ✅ Workflow completed successfully. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Scope of work done
Summary
Adds parallel test sharding to CI, splitting Python unit tests and integration tests across 4 Cloud Build shards running concurrently. Slow
test modules are explicitly pinned to specific shards via ordered tuples, preventing hash-collision hotspots. Unpinned modules are assigned via
SHA-256 hash.
and reports pass/fail with build URLs
new CLI args --shard_index/--total_shards
measured durations as comments
that scan tests/unit/ to verify sharded == unsharded
As a followup, we should probably move that inlined bash script to be python but then we'd need to add some new
cipython deps that we just install for ci purposes and that seems like a different change.NOTE: I tested this in a slightly different
on: pushpath, I'll test this change when it gets merged and if the ci setup breaks I can revertWhere is the documentation for this feature?: N/A
Did you add automated tests or write a test plan?
Updated Changelog.md? NO
Ready for code review?: NO