chore(migration): Migrate code from googleapis/python-pubsub into packages/google-cloud-pubsub

[parthea]

See #10930.

This PR should be merged with a merge-commit, not a squash-commit, in order to preserve the git history.

[gemini-code-assist]

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request completes a significant migration of the Google Cloud Pub/Sub Python client library's codebase into a dedicated packages/google-cloud-pubsub directory. This restructuring involves comprehensive updates to build, testing, and CI/CD configurations to support the new package layout. Additionally, it integrates OpenTelemetry tracing capabilities across both publishing and subscribing functionalities, enhancing observability. The changes also include various refinements to message handling, flow control, and sequencing mechanisms within the client library.

Highlights

• Codebase Migration: The entire codebase for the Google Cloud Pub/Sub Python client library has been migrated into a new packages/google-cloud-pubsub directory, aligning with a monorepo structure.
• Configuration Updates: Numerous configuration files, including .coveragerc, .flake8, .github workflows (for docs, lint, unittest), .kokoro CI/CD configurations, and .librarian generator inputs, have been added or updated to support the new package structure and development processes.
• OpenTelemetry Tracing: New OpenTelemetry tracing support has been introduced for both publisher and subscriber clients, enabling context propagation and span management for message creation, flow control, batching, and subscriber operations (ack/nack/modack).
• Publisher and Subscriber Enhancements: Refinements were made to publisher batching, flow control, and message sequencing logic, including handling of ordered messages and error conditions. Subscriber-side protocol elements like dispatcher, leaser, and streaming pull manager were also updated.
• Documentation and Metadata: Project documentation (CHANGELOG.md, CONTRIBUTING.rst, README.rst, SECURITY.md, UPGRADING.md) and repository metadata (.repo-metadata.json) have been updated to reflect the new package location and current development guidelines.

Changelog

• packages/google-cloud-pubsub/.librarian/generator-input/librarian.py
  • Updated Python script for library generation logic, including gRPC options, emulator support, deprecation warnings, and OpenTelemetry integration.
• packages/google-cloud-pubsub/.librarian/generator-input/noxfile.py
  • Updated Nox configuration for testing, linting, and documentation generation.
• packages/google-cloud-pubsub/.librarian/generator-input/setup.py
  • Updated setup.py configuration for package metadata and dependencies.
• packages/google-cloud-pubsub/CHANGELOG.md
  • Updated the changelog to reflect recent releases and features.
• packages/google-cloud-pubsub/google/cloud/pubsub_v1/publisher/_batch/thread.py
  • Updated thread-based implementation for Pub/Sub message batching to include OpenTelemetry spans.
• packages/google-cloud-pubsub/google/cloud/pubsub_v1/publisher/_sequencer/ordered_sequencer.py
  • Updated ordered sequencer for Pub/Sub messages to pass OpenTelemetry data.
• packages/google-cloud-pubsub/google/cloud/pubsub_v1/publisher/_sequencer/unordered_sequencer.py
  • Updated unordered sequencer for Pub/Sub messages to pass OpenTelemetry data.
• packages/google-cloud-pubsub/google/cloud/pubsub_v1/publisher/client.py
  • Updated Publisher client implementation with OpenTelemetry tracing, batching, and flow control.
• packages/google-cloud-pubsub/google/cloud/pubsub_v1/publisher/flow_controller.py
  • Updated flow controller for managing message publishing limits.
• packages/google-cloud-pubsub/google/cloud/pubsub_v1/subscriber/_protocol/dispatcher.py
  • Updated dispatcher for handling subscriber requests, including OpenTelemetry span management for ack/nack/modack.
• packages/google-cloud-pubsub/google/cloud/pubsub_v1/subscriber/_protocol/leaser.py
  • Updated leaser for managing message leases, including OpenTelemetry data handling.
• packages/google-cloud-pubsub/google/cloud/pubsub_v1/subscriber/_protocol/messages_on_hold.py
  • Updated mechanism for tracking messages on hold by ordering key to include OpenTelemetry spans.
• packages/google-cloud-pubsub/google/cloud/pubsub_v1/subscriber/_protocol/requests.py
  • Updated named tuples for subscriber management requests to include OpenTelemetry data.
• packages/google-cloud-pubsub/google/cloud/pubsub_v1/subscriber/_protocol/streaming_pull_manager.py
  • Updated streaming pull manager for coordinating message consumption, including OpenTelemetry span management.
• packages/google-cloud-pubsub/google/cloud/pubsub_v1/subscriber/client.py
  • Updated Subscriber client implementation with OpenTelemetry tracing, streaming pull, and flow control.
• packages/google-cloud-pubsub/google/cloud/pubsub_v1/subscriber/message.py
  • Updated Message class for representing Pub/Sub messages, including OpenTelemetry data.
• packages/google-cloud-pubsub/google/cloud/pubsub_v1/subscriber/scheduler.py
  • Updated scheduler for asynchronous callback execution.
• packages/google-cloud-pubsub/google/cloud/pubsub_v1/types.py
  • Updated custom types and flow control settings for Pub/Sub clients, including OpenTelemetry tracing option.
• packages/google-cloud-pubsub/google/pubsub/gapic_version.py
  • Updated GAPIC version to 2.35.0.
• packages/google-cloud-pubsub/google/pubsub_v1/init.py
  • Updated __init__.py for pubsub_v1 package, including version checks and dependency warnings.
• packages/google-cloud-pubsub/google/pubsub_v1/gapic_version.py
  • Updated GAPIC version to 2.35.0.
• packages/google-cloud-pubsub/google/pubsub_v1/services/publisher/async_client.py
  • Updated asynchronous Publisher client.

Activity

• This pull request is part of a larger migration effort, referencing issue Adopt split repo: _python-pubsub_ #10930.
• The author has specified that this PR should be merged with a merge-commit to preserve git history.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request is a large migration that moves the google-cloud-pubsub package into this repository, adding a significant number of new configuration and source files. My review focused on these new files. I've identified a critical issue in the setup.py file concerning a non-existent dependency version which will break installation. Additionally, there are a few medium-severity issues related to outdated information in documentation (which is now linked to a rule about environment breaking changes), dead code, and a fragile implementation that relies on internal details of a standard library module. Addressing these points will improve the quality and robustness of the migrated package.

_{Note: Security Review is unavailable for this PR.}

[gemini-code-assist]

The grpcio version 1.75.1 specified for python_version >= '3.14' does not appear to be a valid published version. This will likely cause installation failures for users on Python 3.14 and newer. Please verify and correct the required grpcio version.

[gemini-code-assist]

This document, which is part of the current version of the library, states that the 2.0.0 release requires Python 3.6+. However, this library version requires Python 3.9+. This could be confusing for users. Please consider updating this to reflect the current minimum version requirement.

Suggested change

	The 2.0.0 release requires Python 3.6+.
	The 2.0.0 release requires Python 3.9+.

References

1. When a release introduces breaking changes in environment requirements, such as dropping support for specific Python versions or major dependency versions (e.g., protobuf), prefer a minor version bump over a patch version bump. This maintains the ability to provide patches for the previous minor version for users still requiring the older environment.

[gemini-code-assist]

This version check for Python < 3.8 seems to be dead code, as the package requires Python 3.9 or newer according to setup.py. This block can be safely removed to improve code clarity.

[gemini-code-assist]

Similar to the publisher client, this version check for Python < 3.8 appears to be dead code since the library requires Python 3.9+. Removing this block would be a good cleanup.

[gemini-code-assist]

This implementation relies on accessing the private _work_queue of ThreadPoolExecutor, which is fragile and could break in future Python versions. Since this library requires Python 3.9+, you could consider using the cancel_futures=True argument in executor.shutdown(), which was introduced in Python 3.9. To maintain the contract of returning dropped messages, you might need to maintain a separate collection of pending futures within the ThreadScheduler and, upon shutdown, cancel them and extract the message arguments from the futures that were not started.