Support for running hero workloads

[mimowo]

What would you like to be added:

Dedicated API, likely a dedicated Hero/Uber ClusterQueue type/mode, for running "hero" workloads.

There is no one definition of a hero workload, but they have common characteristics: they often take >50% of the cluster capacity and run for a prolonged time (weeks), and are "super high priority". Such workloads are often used for AI training, and basically impact the cluster for the entire organization within the period.

Why is this needed:

There is currently no go-to setup for running "hero" workloads.

For example, getting suddenly >50% of cluster quota is hard, because even if the workload has "super high priority", then it still cannot preempt from CQs which are below nominal quota.

Different users make different approaches which have their pros / cons, and eventually come up with something that "works", but it is re-discovering the wheel.

Completion requirements:

This enhancement requires the following artifacts:

• Design doc
• API change
• Docs update

The artifacts should be linked in subsequent comments.

[mimowo]

/assign @mwielgus
who I think is currently cooking some design
cc @tenzen-y @gabesaba @sohankunkerkar @kannon92 @mwysokin

[mimowo]

Note that it might be considered an alternative design for: #6141 (at least with strong overlap for use-cases). I tentatively think this approach per CQ is more aligned with the principles of Kueue, but let's re-evaluate both designs when posted.

[tenzen-y]

Note that it might be considered an alternative design for: #6141 (at least with strong overlap for use-cases). I tentatively think this approach per CQ is more aligned with the principles of Kueue, but let's re-evaluate both designs when posted.

As I checked #6141 #8654, this doesn't cover the #6141 #8654 use case. It seems that they want to relax quotas in a specific tenants managed by a single large shared ClusterQueue.

So, I think this request could be considered another one, and totally agree with the described story in this issue.

[EDIT] I left the above comment assuming #8654. I think that this could replace #6141, surely. Let's see which (#6141 vs this) can cover the almost stories.