33.6 切换与重建 runbook

step: stable-id
intent: why
preconditions: [...]
command_or_action: exact
risk_class: R0 | R1 | R2 | R3
expected: [...]
evidence: [...]
stop_if: [...]
rollback_or_next_safe_state: [...]
owner: role

healthy leader + maintenance need
  -> planned switchover

leader process demonstrably failed/fenced
  + automatic policy healthy
  -> observe automatic failover

no accepted leader
  + automatic path cannot complete
  + exact candidate/fence/data risk approved
  -> manual failover

replica broken, leader healthy
  -> restart/rewind/reinit replica; do not fail over

pig pt list pg-test -o json
pig pt config show -o json

SELECT pg_is_in_recovery(),
       (pg_control_system()).system_identifier,
       (pg_control_checkpoint()).timeline_id;

one primary / expected replicas
pause=false
member tags
ttl / loop_wait / retry_timeout
maximum_lag_on_failover
synchronous and failsafe modes
watchdog mode
sender/receiver and replay gap
DCS member/quorum
service path and client probe

pig pt switchover --plan
pig pt switchover \
  --leader <current> \
  --candidate <target>

new timeline
old leader streaming
client writes on new authority
unknown outcome reconciled
archive follows new primary
scheduled jobs no duplicate

pig pt failover --candidate <exact-member> --plan

old-primary fence
candidate lineage and lag
last ack / unknown manifest
accepted data loss
DCS authority
business owner and incident commander authorization

pig pt reinit <replica> --plan

DCS leader = A
Patroni REST A = replica
SQL A recovery = true
SQL B recovery = false

capture before
start client probe
record action monotonic timestamp
perform one bounded action
capture fence and transition
capture after
reconcile client identities
only then perform cleanup/baseline restore
capture restored

scope / member
DCS kind and endpoint count
config booleans and numeric policy
credential present + hash/length where necessary
REST role/state/timeline
raw log hash + selected event counts

started_at UTC
started_monotonic_ns
finished_at UTC
finished_monotonic_ns

requirements + failure model
before phase
fault action
old-primary fence
failed phase + selected candidate
rejoin phase
client event stream + reconciliation
baseline restore action + restored phase
DCS tabletop decision
rewind/basebackup evidence
cleanup
source hashes
positive validation + adversarial mutations
public summary

database authority
replication and slots
client route and unknown outcomes
WAL archive
backup schedule and repository
monitoring/alerts
jobs, CDC, logical replication
capacity and cache
security/audit

SELECT application_name, state, sync_state, replay_lsn
FROM pg_stat_replication;

SELECT slot_name, active, restart_lsn, wal_status,
       safe_wal_size, invalidation_reason
FROM pg_replication_slots;

pg_stat_archiver last success/error
pgBackRest stanza status/check
new-primary archive command and credentials
latest backup lineage/timeline
repository capacity and retention
next scheduled backup owner

timeline increase
backend PID reset
cumulative stats reset/restart
cache hit ratio temporarily下降
replica lag spike
connection errors spike
checkpoint/archive timing change

authority:
  unique_primary: true
  old_primary: streaming_or_decommissioned
data:
  acknowledged_missing: 0
  unknown_unreconciled: 0
  accepted_loss: documented
replication:
  expected_members_streaming: true
  slots_reconciled: true
service:
  write_route_valid: true
  stale_connections_drained: true
recoverability:
  archive_healthy: true
  backup_healthy: true
  restore_followup_scheduled: true
operations:
  alerts_restored: true
  jobs_and_cdc_validated: true
  evidence_bundle_sealed: true
business:
  owner_acceptance: approved | pending