[OPIK-7231] [BE] feat: ingest cipx spend blocks into dedicated table with derived allocation

[BorisTkachenko]

Details

Corresponding retrieval path changes: https://github.com/comet-ml/ai-cost-backend/pull/8

Moves cipx block data out of cipx_spends.blocks (an Array(Tuple) holding ~99% of the row
weight that every composition/breakdown query had to ARRAY JOIN and re-derive with window
functions) into a dedicated cipx_spend_blocks table with one row per block. Token allocation
and dashboard groupings (tier, lane, bd_lane, label, is_definition, alloc) are now
derived once in Java at ingestion time, so retrieval becomes plain GROUP BYs.

• New cipx_spend_blocks table (migration 000098) fed by CostIntelligenceIngestionListener
  on span create events; span ingestion is create-only (cipx call data is immutable), so the
  SpanCostIntelligenceChanged span-update event is removed.
• cipx_spends drops the blocks column (000097); cipx_trace_identities is recreated with
  user_uuid in the primary key (000099); new MySQL cipx_user_mappings email -> uuid lookup
  table (000088) populated from trace identity events.
• Blocks are bulk-inserted via the ClickHouse v2 HTTP client (JSONEachRow, one HTTP body). One
  span fans out to ~350 block rows, and the R2DBC driver resolves each named bind with a linear
  scan over the statement's parameter list — a 200-span batch is ~1.7M parameters, which is
  O(n²) (hours of CPU) and wedged the event-bus subscriber. The remaining cipx DAOs
  (CipxSpendDAO, CipxTraceIdentityDAO) switch to O(1) positional binds, and the listener
  logs subscribe/complete per insert (cipx-bearing events only) for ingestion visibility.

Change checklist

• User facing
• Documentation update

Issues

• Resolves #
• OPIK-7231

AI-WATERMARK

AI-WATERMARK: yes

• Tools: Claude Code
• Model(s): Claude Fable 5
• Scope: ingestion-path performance diagnosis (thread dumps, ClickHouse query_log analysis) and
  the resulting fixes: v2-client bulk insert for cipx_spend_blocks, positional binds in the
  cipx DAOs, ingestion visibility logs in the listener; test data replay scripts.
• Human verification: reviewed all changes; validated end-to-end locally (see Testing).

Testing

• CostIntelligenceIngestionTest covers the listener end-to-end: cipx span create ->
  cipx_spends + cipx_spend_blocks rows (attributed + residual, allocation math), trace
  create/update -> cipx_trace_identities + cipx_user_mappings.
• Validated with real data by replaying a staging dataset (16,281 cipx spans / 14,492 traces,
  original IDs preserved) through the local BE REST endpoints
  (/v1/private/{traces,spans}/batch, batches of 200):
  • 100% span parity: 16,281/16,281 cipx_spends rows, 5.98M cipx_spend_blocks rows
    (~367 blocks/span), 14,490 trace identities; 0 errors.
  • Before the bulk-insert fix, the first 68,812-row blocks insert ran for 2+ hours holding the
    event-bus subscriber monitor (only 202 of 16,281 spans landed); after the fix the same
    insert completes in ~340 ms.
  • Dashboard totals match staging (the remaining 50-span delta is a pre-existing staging-side
    ingestion gap: its cipx_spends is missing 50 spans that exist in its raw spans table).

Documentation

N/A — schema and derivation rules are documented in the migration files and DAO javadocs.

[github-actions]

⏱️ pre-commit per-hook timing

Hook	Description	Result	Duration
☕ spotless — java backend	Format Java code	✅	4.08s
Total (1 ran)			4.08s

⏭️ 39 skipped (no matching files changed)

Hook	Description	Result
🐍 trim trailing whitespace — python sdk	Strip trailing whitespace	⏭️
🐍 fix end of files — python sdk	Ensure files end in a newline	⏭️
🐍 ruff — python sdk	Lint + autofix Python (ruff)	⏭️
🐍 ruff-format — python sdk	Format Python code (ruff)	⏭️
🐍 mypy — python sdk	Static type check	⏭️
🤖 trim trailing whitespace — optimizer	Strip trailing whitespace	⏭️
🤖 fix end of files — optimizer	Ensure files end in a newline	⏭️
🤖 check yaml — optimizer	Validate YAML syntax	⏭️
🤖 check json — optimizer	Validate JSON syntax	⏭️
🤖 check toml — optimizer	Validate TOML syntax	⏭️
🤖 check for added large files — optimizer	Block large files (>1MB)	⏭️
🔐 detect private key — optimizer	Block committed private keys	⏭️
🤖 check for merge conflicts — optimizer	Block merge-conflict markers	⏭️
🤖 check for case conflicts — optimizer	Block case-only name clashes	⏭️
🤖 pyupgrade — optimizer	Modernize Python syntax	⏭️
🤖 ruff — optimizer	Lint + autofix Python (ruff)	⏭️
🤖 ruff-format — optimizer	Format Python code (ruff)	⏭️
🤖 mypy — optimizer	Static type check	⏭️
📓 nbstripout — optimizer notebooks	Strip notebook output	⏭️
📝 markdownlint — optimizer	Lint Markdown	⏭️
🔤 codespell — optimizer	Fix common misspellings	⏭️
📊 radon cc — optimizer	Cyclomatic-complexity gate	⏭️
📊 radon raw — optimizer	Raw size metrics gate	⏭️
📊 xenon — optimizer	Fail on complexity thresholds	⏭️
📊 lizard — optimizer	Cyclomatic-complexity gate	⏭️
🧹 vulture — optimizer	Find dead code	⏭️
🛡️ trim trailing whitespace — guardrails	Strip trailing whitespace	⏭️
🛡️ fix end of files — guardrails	Ensure files end in a newline	⏭️
🛡️ ruff — guardrails	Lint + autofix Python (ruff)	⏭️
🛡️ ruff-format — guardrails	Format Python code (ruff)	⏭️
🛡️ mypy — guardrails	Static type check	⏭️
⚓ helm-docs	Regenerate Helm chart README	⏭️
block non-public FE plugins	Block non-public FE plugins	⏭️
🧪 pre-commit wrapper smoke tests	Self-test the wrapper scripts	⏭️
🌐 eslint — frontend	Lint + autofix JS/TS	⏭️
🌐 typecheck — frontend	Whole-project tsc type check	⏭️
📘 eslint — typescript sdk	Lint + autofix JS/TS	⏭️
📘 typecheck — typescript sdk	Whole-project tsc type check	⏭️
⚙️ actionlint — github workflows	Lint GitHub Actions workflows	⏭️