20.6 交付并观察 HA 集群
Pigsty 把 PostgreSQL、Patroni、etcd、HAProxy、PgBouncer、监控与配置交付 组合起来。平台的价值不是隐藏原理,而是让同一 HA 合同可以声明、部署、 观察和重复执行。
本节坚持两条线同时存在:
Pigsty declaration and operator entry
PostgreSQL/Patroni native evidence平台显示与原生事实不一致时,不选一个“更顺眼”的相信,而是停止并解释 差异。
20.6.1 拓扑、同步策略与服务端点声明
第 19 章保留的 service unit
pg-test-1 10.10.10.11 declared primary
pg-test-2 10.10.10.12 declared replica
pg-test-3 10.10.10.13 declared replica + offline intent关键点是声明 stable identity 与 placement intent,不把 primary 当成永远
属于某个 host 的固定属性。Patroni 运行时可以改变 role。
一个简化、无 credential 的结构示意:
pg-test:
vars:
pg_cluster: pg-test
pg_version: 18
pg_conf: crit.yml
hosts:
10.10.10.11:
pg_seq: 1
pg_role: primary
10.10.10.12:
pg_seq: 2
pg_role: replica
10.10.10.13:
pg_seq: 3
pg_role: offline这不是正式 live inventory;具体 schema 以所用 Pigsty release 和 Cluster / Instance 文档 为准。真实 inventory 可能含 credential,只能保存在 private mode-0600 文件中。
declaration 的边界
inventory 能说明:
desired membership
stable instance identity
initial placement intent
parameter template
service definition不能单独证明:
live PostgreSQL role
replication caught up
DCS authority
client routing
failure-domain independence
RPO/RTO所以第 19 章 acceptance 与本章 phase capture 都要读 live state。
同步策略有两处 authority
区分:
local Patroni config
member-specific bootstrap/connectivity/watchdog/DCS endpoint
dynamic Patroni config in DCS
ttl, loop_wait, sync mode, failover lag, PostgreSQL parameters修改 DCS dynamic config 后,只查 inventory 会读到旧意图;只查某台本地 YAML 也可能漏掉 cluster-level state。
本章 capture:
each member local:
scope/member_name/version
dcs kind and endpoint count
watchdog mode
REST/PostgreSQL connect address
cluster dynamic:
ttl/loop/retry
maximum_lag_on_failover
sync modes
pause/failsafe
use_pg_rewind/use_slots完整 config 可能含 secret,evidence 只导出 allowlist。
service 是对外能力,不是节点别名
Pigsty 默认服务抽象:
primary :5433
read-write -> current primary -> default target usually PgBouncer
replica :5434
read-only -> eligible replicas; policy can define fallbacks
default :5436
admin/direct -> current primary PostgreSQL
offline :5438
offline/OLAP placement服务定义包含:
port
destination: pgbouncer/postgres
Patroni health endpoint
member selector
backup selector参考当前 Pigsty Service/Access。
primary service 的数据路径
默认可概括:
client
-> member/VIP/DNS :5433
-> HAProxy
-> backend health on Patroni :8008 /primary
-> current primary PgBouncer :6432
-> PostgreSQL :5432每一跳都可能影响恢复时间。HAProxy 看到新 primary,不代表 pool 中每条旧 connection 都可继续。
endpoint 也要版本化
服务合同应记录:
name/port/protocol
write/read semantics
pooling mode
TLS identity
health source
fallback selectors
timeouts
max connections/queue
DNS/VIP provider
owner端口号相同不等于 release 间行为完全相同。升级时应 diff render 后的 HAProxy/PgBouncer/Patroni config。
offline 不是“慢查询免疫”
pg-test-3 的 offline intent 能影响 service selector;但它仍:
- 共享同一 WAL history;
- 竞争主机 CPU/memory/storage;
- 可能因长查询产生 recovery conflict;
- 在本沙箱共享 hypervisor;
- 不是 delayed backup。
placement label 必须由 metrics 和 workload policy 验证。
配置同步复制前
不要直接修改一条参数。先形成 decision:
failure scope
acknowledgement class
FIRST/ANY set
candidate tags
strict/degrade behavior
placement
latency budget
test plan
rollbackPigsty/Patroni 是交付入口,PostgreSQL commit semantics 仍按 20.3 解释。
20.6.2 从 Patroni、SQL 和指标验证角色
第一层:Patroni topology
当前 Pigsty 提供:
pig pt list pg-test
pig pt config show
pig pt status或原生:
patronictl -c /etc/patroni/patroni.yml \
list pg-test --format=json
patronictl -c /etc/patroni/patroni.yml \
show-config pg-test看:
cluster/member identity
one leader
replica state
timeline
lag
pause
dynamic policy正式实验使用 exact v4.4.0 部署内的原生 patronictl,避免让后来更新的
wrapper 行为被冒充为当时执行路径。正文同时介绍当前 pig pt,但保留版本
边界。
第二层:SQL role
每个 member:
SELECT pg_is_in_recovery(),
current_setting('cluster_name'),
current_setting('server_version_num');预期:
one false -> current primary
two true -> standbys
cluster_name = pg-test
server major = 18若 Patroni 说 primary、SQL 却 pg_is_in_recovery()=true,不要把它当成
“几秒后会好”直接继续 mutation。
第三层:replication direction
primary:
SELECT application_name, client_addr, state, sync_state,
sent_lsn, flush_lsn, replay_lsn
FROM pg_stat_replication;standby:
SELECT status, sender_host, sender_port,
written_lsn, flushed_lsn, latest_end_lsn
FROM pg_stat_wal_receiver;正式 validator 不只检查 row count,还检查:
application names = exact nonleaders
client address = declared address
receiver upstream = exact current primary
state = streaming第四层:lineage
SELECT system_identifier FROM pg_control_system();
SELECT timeline_id FROM pg_control_checkpoint();primary 另外从 current WAL filename 得到 current timeline。
判定:
same system identifier all members/all phases
one Patroni timeline per phase
primary current WAL timeline equals Patroni
timeline advances on promotion
checkpoint timeline is named and interpreted correctly第五层:retention
SELECT slot_name, active, restart_lsn, wal_status, safe_wal_size
FROM pg_replication_slots;stable primary 的 slot 必须与两个 replica 一一对应。另看:
pg_wal filesystem
archive success/failure
WAL generation rateslot active 不是“永远安全”,只说明 consumer 当前使用。
第六层:service
从 client path:
psql "service=pg36-ch20" -X -w \
-c "select pg_is_in_recovery(), inet_server_addr();"不要打印 service file;它含 password。正式 helper 只输出:
status=private-service-created
secret_values_exported=0client path 要验证:
connect
read-write attribute
server role
transaction
reconnect through transition第七层:metrics 与 logs
观察面板/指标至少覆盖:
Patroni member/leader changes
WAL generation/send/receive/replay
replication lag
slots/WAL retention
HAProxy backend state and sessions
PgBouncer connections/wait
PostgreSQL transaction/lock/error
host CPU/memory/disk/network
DCS latency/health但 dashboard 颜色仍要回到 query definition。metric label primary 是从谁
推导的?采样周期多长?切换时有没有 stale series?
角色一致性矩阵
| phase | Patroni leader | SQL primary | service write target | upstream |
|---|---|---|---|---|
| before | pg-test-1 | .11 | .11 pool | replicas←.11 |
| forward | pg-test-2 | .12 | .12 pool | replicas←.12 |
| restored | pg-test-1 | .11 | .11 pool | replicas←.11 |
四列不能只靠一份 patronictl list 填满。
capture hygiene
正式采集固定:
SSH -F /dev/null
BatchMode=yes
allowlisted local config
no arbitrary Patroni tags
no credential output
structured JSON
source SHA-256禁用本机 SSH config 是因为第 19 章曾发现地址 alias/forwarding 会把多个
目标看成同一 guest。生产不能照抄 StrictHostKeyChecking=no;本选择只为
disposable local sandbox。
20.6.3 演练动作对应的 Pigsty 入口与原生证据
当前 Pigsty 操作入口
在当前文档版本:
pig pt list pg-test
pig pt switchover --plan
pig pt switchover -l pg-test-1 -c pg-test-2
pig pt failover -c pg-test-2 --plan
pig pt config show
pig pt log -fpig pt 封装常见 patronictl/systemctl 操作。switchover 是计划切换;
failover 是不健康 cluster 的 manual failover,不能互换。
reinit 会删除目标 member 数据并重新同步,是破坏性动作:
pig pt reinit pg-test-2 --plan本章不执行 reinit。
平台入口与原生命令对照
| 意图 | Pigsty 当前入口 | 原生核心 | 证据 |
|---|---|---|---|
| 列成员 | pig pt list | patronictl list | JSON + SQL |
| 看策略 | pig pt config show | patronictl show-config | DCS config |
| 计划切换 | pig pt switchover | patronictl switchover | timeline/client |
| 手工故转 | pig pt failover | patronictl failover | data risk/fence |
| 重加成员 | pig pt reinit | Patroni reinit | base copy/rejoin |
| 服务路径 | rendered service | HAProxy/Patroni/PgBouncer | external probe |
wrapper 改善 ergonomics 与 preflight,不改变底层状态迁移的风险等级。
本章为什么用原生 patronictl
正式 target 是 exact Pigsty v4.4.0 archive。实验记录必须说明当时实际可用 的执行机制:
executor=patronictl
config=/etc/patroni/patroni.yml
cluster=pg-test
leader/candidate explicit当前 pig pt 文档在书写时已经提供更完整 wrapper;它适合读者检查当前
环境,但不能篡改历史 evidence。
正常 all 为什么不调用 switchover
一个危险的工具设计:
task.sh all
-> capture
-> switchover
-> verify用户可能只想重验报告,却意外移动 primary。
本章语义:
capture read-only current snapshot
verify validate retained evidence
review provenance + interpretation
all verify + review only
drill:switchover
separately guarded L2 action
reset:fixture
separately guarded destructive action安全应该体现在 interface,而不只写在注释里。
live drill guard
需要全部精确满足:
PG36_CH20_TARGET=pg36-l2-vagrant/pg-test
PG36_CH20_NONPRODUCTION=true
PG36_CH20_PRODUCTION_DATA=false
PG36_CH20_PRODUCTION_TRAFFIC=false
PG36_CH20_CONFIRM=SWITCH_CH20_PG_TEST_1_TO_2_AND_BACK
new empty evidence directory
private mode-0600 chapter-19 inventory底层 drill.py 再验证:
same exact target/confirmation/authority
service file mode=0600 and not symlink
host/port/database/user/service attributes match contract
output directory emptydefense in depth 防止绕过 wrapper。
preflight 与 postflight
外层动作:
chapter 19 all -> pass
private service generation
chapter 20 drill
positive + ten negative validations
chapter 19 all -> pass
chapter 20 review
temporary secret cleanuppostflight 不是形式主义。它证明:
pg-test-1 returned as unique primary
two replicas stream
host/service baseline not drifted
production gate remains pendingsource identity
manifest 保存所有 decision/executable input 的 SHA-256。review.py 要求当前
source 与 run source 一致。
两份 outcome 文件不作为下一次输入:
drill-run.json
migration-effort.json它们被明确排除 manifest source set,避免“运行结果参与定义自己的输入” 循环;review 仍单独验证其 schema 与解释边界。
反例
正常 report 通过还不够。十个 corruption 必须被拒绝:
claim production from sandbox
switch unreviewed candidate
ignore action failure
foreign system identifier
two leaders
no timeline advance
old primary not rejoined
acknowledged token lost
unknown outcome unreconciled
write gap above objective这使 validator 不只会接受 happy path,也证明关键 guard 真能失败。
本节操作边界
安全 read-only:
export PG36_EVIDENCE_DIR=/private/evidence/ch20-formal
static/labs/ch20/task.sh all不要从书页复制 live mutation 到生产。先读:
小结
inventory declares; live planes prove
service abstracts role, not failure semantics
Pigsty wrappers map to Patroni/PostgreSQL primitives
exact release boundaries matter
all must be safe to repeat
mutation, validation, and reset need separate authority
negative tests protect interpretation权威参考
- Pigsty:Cluster / Instance
- Pigsty:Service/Access
- Pigsty:
pig pt - Pigsty:High Availability
- 本章任务入口
- 本章 evidence review
上一节:切换、故障转移与重加入 · 返回本章目录 · 下一节:实战:一次有证据的计划切换 · 查看全书目录 · 查看索引中心