Correct shutdown behavior for DDBLeaseCoordinator #1700
Conversation
vincentvilo-aws
had a problem deploying
to
manual-approval
GitHub Actions
Failure
vincentvilo-aws
had a problem deploying
to
manual-approval
GitHub Actions
Failure
vincentvilo-aws
had a problem deploying
to
manual-approval
GitHub Actions
Failure
vincentvilo-aws
had a problem deploying
to
manual-approval
GitHub Actions
Failure
vincentvilo-aws
had a problem deploying
to
manual-approval
GitHub Actions
Failure
vincentvilo-aws
had a problem deploying
to
manual-approval
GitHub Actions
Failure
vincentvilo-aws
temporarily deployed
to
manual-approval
GitHub Actions
Inactive
| log.info("Stopping lease taker..."); | ||
| // the method is called in worker graceful shutdown. We want to stop any further lease shutdown | ||
| // so we don't interrupt worker shutdown. | ||
| Optional.ofNullable(leaseGracefulShutdownHandler).ifPresent(LeaseGracefulShutdownHandler::stop); |
There was a problem hiding this comment.
Can we also make sure in GracefulShutdownCallable we are stopping WorkerMetricStatsReporter so that the leader can stop assigning leases to this work once the shutdown has begun ?
There was a problem hiding this comment.
When the scheduler gets shutdown, a shutdown call is made to the initializer here which stops the workerMetricsReporter
There was a problem hiding this comment.
We want to do it in the context of graceful shutdown before lease discoverer is stopped to prevent leader from assigning new leases to this worker which is currently going down. We can make it as a followup change, since this CR itself prevent the worker from picking up leases during the graceful shutdown process so its worth getting this in first.
There was a problem hiding this comment.
3 participants
Issue #, if available:
Description of changes: Moving the
leaseGracefulShutdownHandler.stop()and theleaseDiscoveryFuture.cancel()method outside of the takerFuture condition.By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.