Skip to content

Comments

[SPARK-55610][PYTHON] Add getExecutorInfos to StatusTracker in Python#54389

Open
WweiL wants to merge 2 commits intoapache:masterfrom
WweiL:SPARK-55610-statustracker-executorinfos
Open

[SPARK-55610][PYTHON] Add getExecutorInfos to StatusTracker in Python#54389
WweiL wants to merge 2 commits intoapache:masterfrom
WweiL:SPARK-55610-statustracker-executorinfos

Conversation

@WweiL
Copy link
Contributor

@WweiL WweiL commented Feb 20, 2026

What changes were proposed in this pull request?

This API is not there in Python but has been there in Scala

Why are the changes needed?

Add Python missing API

Does this PR introduce any user-facing change?

Yes, now user can access executor infos in PySpark (non-connect)

>>> spark.sparkContext.statusTracker().getExecutorInfos()
[SparkExecutorInfo(host='lus-j0qh2kkxjt.tailf4adb.ts.net', port=59602, cacheSize=0, numRunningTasks=0, usedOnHeapStorageMemory=0, usedOffHeapStorageMemory=0, totalOnHeapStorageMemory=455501414, totalOffHeapStorageMemory=0)]

How was this patch tested?

Added unit, also manually tested

Was this patch authored or co-authored using generative AI tooling?

The entire PR is written with AI except this PR description

@WweiL WweiL force-pushed the SPARK-55610-statustracker-executorinfos branch from 3a0200f to 772e3e9 Compare February 22, 2026 05:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant