In T423151#11828091, @ops-monitoring-bot wrote:

cookbooks.sre.hosts.decommission executed by marostegui@cumin1003 for hosts: clouddb1019.eqiad.wmnet

• clouddb1019.eqiad.wmnet (FAIL)
  • Host not found on Icinga, unable to downtime it
  • Found physical host
  • Downtimed management interface on Alertmanager
  • Unable to connect to the host, wipe of swraid, partition-table and filesystem signatures will not be performed: Cumin execution failed (exit_code=2)
  • Failed to power off, manual intervention required: Remote IPMI for clouddb1019.mgmt.eqiad.wmnet failed (exit=1): b''
  • [Netbox] Set status to Decommissioning, deleted all non-mgmt IPs, updated switch interfaces (disabled, removed vlans, etc)
  • Configured the linked switch interface(s)
  • Removed from DebMonitor
  • Removed from Puppet server and PuppetDB

ERROR: some step on some host failed, check the bolded items above

This is ready for DC-Ops

cookbooks.sre.hosts.decommission executed by marostegui@cumin1003 for hosts: clouddb1019.eqiad.wmnet

• clouddb1019.eqiad.wmnet (FAIL)
  • Host not found on Icinga, unable to downtime it
  • Found physical host
  • Downtimed management interface on Alertmanager
  • Unable to connect to the host, wipe of swraid, partition-table and filesystem signatures will not be performed: Cumin execution failed (exit_code=2)
  • Failed to power off, manual intervention required: Remote IPMI for clouddb1019.mgmt.eqiad.wmnet failed (exit=1): b''
  • [Netbox] Set status to Decommissioning, deleted all non-mgmt IPs, updated switch interfaces (disabled, removed vlans, etc)
  • Configured the linked switch interface(s)
  • Removed from DebMonitor
  • Removed from Puppet server and PuppetDB

Change #1272222 merged by Marostegui:

[operations/puppet@production] site.pp: Remove clouddb1019

https://gerrit.wikimedia.org/r/1272222

Change #1272222 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] site.pp: Remove clouddb1019

https://gerrit.wikimedia.org/r/1272222

Change #1272216 merged by Marostegui:

[operations/puppet@production] clouddb1019.yaml: Remove file

https://gerrit.wikimedia.org/r/1272216

Change #1272216 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] clouddb1019.yaml: Remove file

https://gerrit.wikimedia.org/r/1272216

Change #1270363 merged by Majavah:

[operations/dns@master] wikimedia.org: Restore original TTL for dumps

https://gerrit.wikimedia.org/r/1270363

I tested one full run on all databases on clouddb1017. It's slower than the old version, but acceptable: it took 12 minutes in total.

Added in https://gerrit.wikimedia.org/r/c/operations/puppet/+/1270891

Change #1270891 had a related patch set uploaded (by FNegri; author: FNegri):

[operations/puppet@production] mariadb: wiki-replicas: add missing grants

https://gerrit.wikimedia.org/r/1270891

Two of the grants are actually needed to create views. This seems to be enough:

These grants are present on all clouddb hosts but are not listed in wiki-replicas.sql:

Change #1270868 merged by Marostegui:

[operations/puppet@production] check_private_data_report: Remove clouddb1019

https://gerrit.wikimedia.org/r/1270868

Change #1270868 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] check_private_data_report: Remove clouddb1019

https://gerrit.wikimedia.org/r/1270868

Change #1270764 merged by Marostegui:

[operations/puppet@production] eqiad.yaml: Remove clouddb1019

https://gerrit.wikimedia.org/r/1270764

Change #1270764 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] eqiad.yaml: Remove clouddb1019

https://gerrit.wikimedia.org/r/1270764

Change #1270758 merged by Marostegui:

[operations/puppet@production] installserver: Remove clouddb1019

https://gerrit.wikimedia.org/r/1270758

Change #1270758 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] installserver: Remove clouddb1019

https://gerrit.wikimedia.org/r/1270758

Re-claiming this task, I'll start with clouddb1022 then.

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1003 for host clouddb1019.eqiad.wmnet with OS trixie executed with errors:

• clouddb1019 (FAIL)
  • Removed from Puppet and PuppetDB if present and deleted any certificates
  • Removed from Debmonitor if present
  • Forced PXE for next reboot
  • Host rebooted via IPMI
  • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console clouddb1019.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.

sounds good from my side yes!

should we go back to reimage clouddb1015