[EKS] [Enhancement]: EKS Upgrade Insights: False positives persist for 30 days after fixing deprecated APIs

[edwanyoike]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Tell us about your request

Enhance EKS Upgrade Insights to validate the current state of resources flagged for deprecated API usage, rather than relying solely on the 30-day audit log window.

Which service(s) is this request for?

EKS

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?

Currently, EKS Upgrade Insights scans cluster audit logs using a 30-day rolling window to detect deprecated API usage. This creates a significant problem for automated upgrade workflows:

1. When a deprecated API call is detected, it's flagged in Upgrade Insights
2. Even after fixing the resource to use the current API version, the error persists in Upgrade Insights for up to 30 days (until the original audit log entry falls outside the rolling window)
3. This blocks automated cluster upgrades that rely on Upgrade Insights for validation, even though the actual resources are using supported API versions

Impact:

• Forces unnecessary use of --force flag in automated upgrades
• Reduces reliability of automation workflows
• Creates false positives that need manual verification
• Could lead to skipping important upgrade readiness checks due to reliance on --force

Are you currently working around this issue?

Currently, we have two suboptimal workarounds:

1. Wait 30 days after fixing deprecated APIs before attempting upgrades
2. Use the --force flag to bypass Upgrade Insights validation, which defeats the purpose of having the validation

Additional context

Proposed enhancement:
When Upgrade Insights detects a deprecated API usage from audit logs, it should:

1. Check the current state of the flagged resource
2. If the resource now uses a supported API version, remove the error from Upgrade Insights
3. Keep the historical audit log data for reference but don't block upgrades based on resolved issues

This would maintain the value of Upgrade Insights while eliminating false positives that currently force the use of --force flag or 30-day waiting periods.

Attachments

N/A

References

1. https://docs.aws.amazon.com/eks/latest/userguide/update-cluster.html#update-cluster-control-plane

[mikestef9]

This was an intentional (although debated) product decision. We decided the only foolproof way to safely detect deprecated API usage in Kubernetes is a rolling look back window through audit logs. Looking at resources inside the cluster (which tools like kube-no-trouble do) is not necessarily good enough, because K8s automatically converts resources in the cluster to newer API versions, and this would also not catch controllers running in the cluster calling deprecated APIs directly.

Take the example of a deployment system that deploys every 15 days or so. On day 2, you might think you have removed all usage of deprecated APIs. You upgrade the cluster. Then 13 days later your deployment system deploys and you get a hard failure. The biggest risk with deprecated API usage is not existing resources in the cluster (because K8s generally auto converts those resources to new API versions). It’s anything that gets deployed after the cluster is upgraded.

After this research and debate, we settled on 30 days as the sweet spot of the time window to look back on (FWIW other cloud providers with similar features have chosen the same window).

A refresh API for upgrade insights would help with checks like addon compatibility, which can be known to be fixed right away. But this wouldn't help with deprecated API usage where we always look back 30 days. One of the use cases of the new force flag, is to upgrade when you are confident you have addressed deprecated API usage, but your cluster hasn't rolled out of our 30 day look back window.

Note that, the upgrade insights feature does surface the last time the deprecated API was called. So each day the upgrade insights refresh runs, that last request time should get an older and older timestamp if that deprecated API is no longer being used, giving you the confidence to upgrade even though the insight is still tagged as an ERROR status.

The only possible feature that could help here is allowing a user defined lookback window, instead of the hardcoded 30 day window. But setting this value to too low of a number runs the deployment use case risk explained above, and I'm not convinced it's control we want to give to users.

[toVersus]

However, some CD tools (e.g. Argo CD) dynamically retrieve available API resources within the cluster using the Discovery API and make API calls accordingly. GKE also has a feature that excludes clusters from automatic updates if they call deprecated APIs. In the past, this caused the issue when clusters stopped updating automatically. See argoproj/argo-cd#9786 (comment).

In the case of EKS, even if you try to update manually, the default behavior prevents updates if there are deprecated API calls in the cluster. The only way to eliminate deprecated API calls is to stop Argo CD, which is not acceptable. As a result, specifying the --force option becomes mandatory for clusters where such tools are used. While this feature might be useful for users who don’t have tools relying on the Discovery API, for some, it simply adds an additional required configuration for the cluster upgrade. I’m not making any specific requests, but I just want it to be understood that this doesn’t necessarily make cluster updates completely safe.