bld: Fix GitHub gpu ci#5107
Open
OliverBryant wants to merge 29 commits into
Open
Conversation
Contributor
|
Note Gemini is unable to generate a review for this pull request due to the file types involved not being currently supported. |
OliverBryant
force-pushed
the
fix-github-gpu-ci
branch
7 times, most recently
from
3922809 to
5b7a9d0
Compare
OliverBryant
force-pushed
the
fix-github-gpu-ci
branch
from
5b7a9d0 to
8fbc17c
Compare
Collaborator
Author
|
/gemini review |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Code Review
This pull request updates the continuous batching tests in xinference/core/tests/test_continuous_batching.py by using longer, more detailed prompts, specifying a max_tokens limit of 512, and increasing the thread synchronization sleep times from 0.01/0.03 seconds to 0.1 seconds. The reviewer recommends further increasing these hardcoded sleep times to 0.5 seconds to prevent potential test flakiness in CI environments.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
This reverts commit 9f809c4.
Keep the Python 3.12 + conda pynini/onnxruntime-gpu 1.22 base and add the optimizations validated on the previous GPU CI iterations: - changes job now maps changed paths to test groups (llm / embedding / image / audio): a PR touching only one model family runs only that family's GPU tests; shared code, build config, or this workflow still trigger a full run, and docs/UI/other-workflow changes skip GPU CI entirely. Also fall back to a full run instead of failing when the diff base is unusable after a force push. - Restore test_continuous_batching to the GPU suite (it is ignored in the non-GPU jobs, so it was not running anywhere) with the race fix that makes the duplicate-request-id and abort scenarios generate long completions instead of finishing within the client round-trip. - Replace ~50 serial pip installs with batched uv installs (no -U: uv upgrades the whole closure, unlike pip) plus a uv wheel cache, and skip the web UI build via NO_WEB_UI=1. - Cache funasr's ModelScope auxiliary models (~1.5GB; the ModelScope CDN accounts for ~15min per run from US runners) and pin XINFERENCE_MODEL_SRC=huggingface with hf_transfer enabled. - Run one pytest process per selected group instead of one for all 18 files: cold starts drop from 18 to at most 4 while an import error or crash in one family cannot take down the others. - Drop --cov from GPU runs (nothing in the repo consumes coverage.xml), add timeout-minutes to bound billing on hangs, and only run GPU CI for pull requests and pushes to main to avoid double-billing.
Three fixes from the last two GPU runs: - Retry the conda install: conda's repodata sqlite cache sporadically throws 'database is locked' on hosted runners and killed the whole job 53 seconds in. - Pin huggingface_hub<1.0 explicitly. hf_hub 1.0 removed is_offline_mode, which diffusers 0.35 and tensorizer still import; transformers 4.53.2 carries the same cap, the pin documents it and protects the resolve order. - Run the virtualenv-based tests (vllm embedding, qwen3-vl reranker) at the end of the embedding group. Creating a model virtualenv leaks its site-packages into the process environment: in the previous combined run, stable diffusion, jina-clip and bge-m3 all imported transformers from the reranker's virtualenv and failed. Keeping one pytest process per family plus this ordering contains the leak.
Collaborator
Author
|
/gemini review |
Contributor
There was a problem hiding this comment.
Code Review
This pull request updates tests in test_continuous_batching.py to use longer prompts and explicit max_tokens to prevent race conditions during duplicate request and abort testing. Additionally, it relaxes the assertions in test_mlx.py to check for non-empty content rather than specific Chinese characters, making the test more robust. There are no review comments, so no feedback is provided.
The uv rewrite kept failing in the conda step (espeak-ng is not packaged on conda-forge; earlier a transient repodata sqlite lock), so go back to the serial pip install block that has installed successfully on the gpu-t4 runner, with three additions at the end: - espeak-ng from apt (TTS phonemization), - hf_transfer for faster HuggingFace downloads, - final guard pins 'numpy>=2,<2.5' and 'huggingface_hub<1.0', because the serial -U installs ignore upper bounds of already-installed packages: numpy 2.5 breaks numba at runtime and huggingface_hub 1.0 removed is_offline_mode still imported by diffusers 0.35/tensorizer. The pip wheel cache replaces the uv cache accordingly.
… group test_qwen3_vl_engine_params also launches its model in a per-model virtualenv (Qwen3-VL-Embedding-2B), so running it first poisoned the process for test_integrated_embedding the same way the reranker virtualenv did in the combined run: peft resolved transformers from the model virtualenv and BGEM3FlagModel/SentenceTransformer imports failed. integrated_embedding is the only non-virtualenv file in the group, so it runs first. Also keep running the remaining groups when one group fails so a single run reports every failure instead of stopping at the first group.
The last run had all four groups fully green (39 passed, 24 skipped, zero FAILED lines) yet the step exited 1: one pytest process returned a nonzero code without reporting a failing test, i.e. it died during interpreter shutdown after the summary. Log each group's exit code so the next run identifies which group and code before deciding on a mitigation.
Two consecutive runs had all four pytest groups fully green (every group exit code 0, the nonzero-code warning never fired, all group markers flushed in order) yet the step still finished with exit code 1 from the explicit 'exit $status' at the end of the login-shell script. Every other passing step on this runner ends via its last command, so finish with a plain test command instead, and unconditionally echo each group's pytest exit code so any future nonzero code is visible in the log.
xinference redirects MODELSCOPE_CACHE to ~/.xinference/modelscope in get_xinference_home(), so ~/.cache/modelscope is never created and the cache save step warned 'Path Validation Error' on every run — funasr's auxiliary models were re-downloaded from ModelScope each time. Point the cache at the real directory.
Collaborator
Author
|
/gemini review |
Contributor
|
Warning Gemini encountered an error creating the review. You can try again by commenting |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.