reduce startup memory by filtering the informer cache

[tommyzli]

Description:
Reduce startup memory usage by filtering the k8s informer caches

Link to tracking Issue(s):

• Resolves: High operator memory usage during start-up #2808

Testing:
E2es pass on my machine, and i tested it on one of my dev clusters. restarts before:

restarts after:

Documentation:

[github-actions]

E2E Test Results

 38 files  241 suites   2h 43m 53s ⏱️
101 tests  99 ✅ 1 💤 1 ❌
263 runs  257 ✅ 2 💤 4 ❌

Results for commit 3a34b4e.

♻️ This comment has been updated with latest results.

[swiatekm]

As an architectural decision, I dislike having two different clients in a mutator and forcing the implementation to decide which one to use. This should be transparent and part of the client's configuration.

Independently, I'd like to understand which of these changes actually have an impact. Right now, this PR includes a lot of sweeping changes to how the operator interacts with the API Server, but no tests or benchmarks of any kind. Before I approve it, I need to have a lot more confidence that it won't result in major performance regressions.

Ideally, it should be possible to reproduce the conditions leading to the startup memory spike using https://github.com/kubernetes-sigs/kwok and verify what exactly the problem is. The original issue isn't as informative as I'd like on this point.

[tommyzli]

@swiatekm I removed the uncached client. With this change on my test environment I still see higher memory usage on startup than the steady state, but on the order of 10% more instead of 100% more.

Ideally, it should be possible to reproduce the conditions leading to the startup memory spike using https://github.com/kubernetes-sigs/kwok and verify what exactly the problem is. The original issue isn't as informative as I'd like on this point.

that sounds like a good thing to have, though I can't commit that I'll have time to work on this