User Details
- User Since
- Aug 29 2023, 8:30 AM (53 w, 3 d)
- Availability
- Available
- LDAP User
- Arnaudb
- MediaWiki User
- ABran-WMF [ Global Accounts ]
Today
I was aiming on monday indeed, those shouldn't need to be handled in the maintenance window as they are on codfw, or am I mistaken?
That bad reboot seems to stem from a hardware issue:
The system board BP1 PG voltage is within range. Fri Sep 06 2024 10:20:57 The system board BP1 PG voltage is outside of range. Fri Sep 06 2024 10:20:37
Yesterday
d/p nodes are repooling
Wed, Sep 4
d/p hosts are repooling
no DNS modification needed, I'll run sudo dbctl --scope codfw section es1 set-master es2030
$ rg 'es1-master' templates/wmnet 57:es1-master 5M IN CNAME es1027.eqiad.wmnet.
d/p hosts are listed below:
Tue, Sep 3
Mon, Sep 2
es2032 | es1 standalone |
es2031 | es2 standalone |
db2207 | s2 candidate master |
db2138 | s2 |
db2125 | s2 |
db2149 | s3 |
db2206 | s4 |
Fri, Aug 30
After testing cookbooks in T372893:
@Marostegui spotted a SQL syntax issue which I fixed, but it broke CI somehow. and I've spotted a micro case sensitivity issue on MasterUseGTID
Thu, Aug 29
Upon reviewing the last run:
I've added a comment on the last PS to allow people to reproduce the tests.
revert had the same bumps with the same good output:
running the cleanup tasks like T373173 fixes the GTID situation, reverting the topology in the same way with PS43
thanks @Marostegui for the help with the platform!
Wed, Aug 28
I've been able to assert sre.switchdc.databases.prepare logic through and through, it looks sound.
I've had an issue on my testing platform to issue SQL commands to avoid running on get_core_dbs() but on get_dbs() selected hosts instead. I've tried to workaround the implementation with no success. PS22 is my last attempt
I'll move on to sre.switchdc.databases.finalize for now and get back to debugging my issue after that. If I don't manage to debug on my own I'll ping @Volans when he gets back.
@Marostegui this should be the last of this rule spam
@jcrespo both severity levels are now excluding backup hosts from replication threads monitoring. If you want me to add a dedicated set of rules, please let me know!
Tue, Aug 27
thanks! <3
host is repooling slowly, we'll see how it behaves with production traffic. I'll reopen if it fails again
repooling in progress
upgrade went through, sanitizing is in progress
Mon, Aug 26
this task depends on: T373175
you preemptively answered my question! thanks
preparation job with the first few critical instances on the path is done for now. I'll have a few host to move during those weeks but this will be manageable.