I was aiming on monday indeed, those shouldn't need to be handled in the maintenance window as they are on codfw, or am I mistaken?

That bad reboot seems to stem from a hardware issue:

	The system board BP1 PG voltage is within range. 	Fri Sep 06 2024 10:20:57
	The system board BP1 PG voltage is outside of range. 	Fri Sep 06 2024 10:20:37

d/p nodes are repooling

In T374087#10120947, @Ladsgroup wrote:

Replica moves is done in two minutes instead of 25 \o/

d/p hosts are repooling

In T373095#10110436, @ABran-WMF wrote:

[...] I'll double check the DNS indeed great catch!

no DNS modification needed, I'll run sudo dbctl --scope codfw section es1 set-master es2030

$ rg 'es1-master'
templates/wmnet
57:es1-master      5M  IN CNAME    es1027.eqiad.wmnet.

d/p hosts are listed below:

In T372893#10106322, @Marostegui wrote:

This should kept open until we have wiped db2230, otherwise it will be forgotten.

good call, I've taken note of the wipe thing on T373579 so we don't forget when T356053 is done.

In T373095#10110383, @Ladsgroup wrote:

Noting that es2032 is a "perceived master" (since dbctl requires a master) so you can't just depool it. You need to switch it over but since it's RO, it's just the dbctl command (and DNS mayyyybe but I'm not sure).

After testing cookbooks in T372893:
@Marostegui spotted a SQL syntax issue which I fixed, but it broke CI somehow. and I've spotted a micro case sensitivity issue on MasterUseGTID

In T372893#10102685, @ABran-WMF wrote:

In T372893#10102518, @ops-monitoring-bot wrote:

cookbooks.sre.switchdc.databases for the switch from test-s4 to test-s4 started by arnaudb@cumin1002 completed:

• MASTER_FROM db2230.codfw.wmnet MASTER_USE_GTID=slave_pos.
• Failed to enable GTID on db2230.codfw.wmnet, current value: Slave_Pos

it appears that

In T372893#10102444, @ABran-WMF wrote:

I've spotted slight issue on class MasterUseGTID(Enum):: [...]NO = "no"

applies as well on that value

Upon reviewing the last run:

I've added a comment on the last PS to allow people to reproduce the tests.

revert had the same bumps with the same good output:

running the cleanup tasks like T373173 fixes the GTID situation, reverting the topology in the same way with PS43

In T372893#10102448, @ops-monitoring-bot wrote:

cookbooks.sre.switchdc.databases for the switch from test-s4 to test-s4 started by arnaudb@cumin1002 completed:

• test-s4 (FAIL)
  • MASTER_FROM db1125.eqiad.wmnet should be read only
  • MASTER_TO db2230.codfw.wmnet STOP SLAVE.
  • MASTER_TO db2230.codfw.wmnet RESET SLAVE ALL.
  • MASTER_TO db2230.codfw.wmnet has no replication set.
  • MASTER_TO db1125.eqiad.wmnet heartbeat server IDs to delete are: []
  • MASTER_FROM db1125.eqiad.wmnet STOP SLAVE.
  • MASTER_FROM db1125.eqiad.wmnet MASTER_USE_GTID=slave_pos.
  • MASTER_FROM db1125.eqiad.wmnet START SLAVE.
  • Failed to enable GTID on db1125.eqiad.wmnet, current value: Slave_Pos

thanks @Marostegui for the help with the platform!

I've been able to assert sre.switchdc.databases.prepare logic through and through, it looks sound.
I've had an issue on my testing platform to issue SQL commands to avoid running on get_core_dbs() but on get_dbs() selected hosts instead. I've tried to workaround the implementation with no success. PS22 is my last attempt
I'll move on to sre.switchdc.databases.finalize for now and get back to debugging my issue after that. If I don't manage to debug on my own I'll ping @Volans when he gets back.

@Marostegui this should be the last of this rule spam
@jcrespo both severity levels are now excluding backup hosts from replication threads monitoring. If you want me to add a dedicated set of rules, please let me know!

thanks! <3

host is repooling slowly, we'll see how it behaves with production traffic. I'll reopen if it fails again

repooling in progress

upgrade went through, sanitizing is in progress

In T373328#10091744, @Marostegui wrote:

Remember this is not only about upgrading, but also about altering the same failed table across all the wikis this host has. But first it needs to be upgraded

this task depends on: T373175

you preemptively answered my question! thanks

preparation job with the first few critical instances on the path is done for now. I'll have a few host to move during those weeks but this will be manageable.