34.7 平台级流量控制与证据
Pigsty 把 PostgreSQL、Patroni、HAProxy、PgBouncer、监控和配置管理组合成平台。它提供 多个可以减压或隔离的控制点,但不会替操作者判断一致性语义,也不会把一条面板曲线 自动变成根因。
34.7.1 从服务端点隔离批处理和只读流量
端口背后是服务契约
Pigsty 的默认服务意图通常是:
| 服务 | 常见端口 | 默认目标 | 适用工作 |
|---|---|---|---|
| primary | 5433 | 当前主库的 PgBouncer | 短 OLTP 读写 |
| replica | 5434 | 可读节点的 PgBouncer | 可容忍副本语义的读 |
| default | 5436 | 当前主库 PostgreSQL 直连 | 管理、迁移、session-sensitive |
| offline | 5438 | offline/replica PostgreSQL 直连 | 受控 OLAP/ETL |
这些是参考配置,不是 PostgreSQL 固有端口;必须以当前 inventory 与生成配置为准。 HAProxy 通常用 Patroni role endpoint 判后端资格,PgBouncer 再把大量 client connection 映射为受控 server connection。
正确隔离:
OLTP write -> primary pooled service
read-with-staleness-contract -> replica pooled service
long report/ETL -> offline service + separate budget
DDL/admin/session semantics -> controlled direct service不正确隔离:
all SELECT -> replica, regardless of read-your-write
all long queries -> offline, regardless of shared storage/CPU
database slow -> send everything to every replica
primary service full -> bypass PgBouncer through direct servicedirect service 是为需要 session/管理语义的受控客户端准备的路径,不是池满时的逃生 后门。若应用能无治理地从 5433 切到 5436,连接预算就失去意义。
路由变化也要验收数据语义
切到 replica/offline 后检查:
SELECT pg_is_in_recovery(),
current_setting('transaction_read_only'),
pg_last_wal_receive_lsn(),
pg_last_wal_replay_lsn(),
pg_last_xact_replay_timestamp();还要用业务 token 验证 staleness/可见性。transaction_read_only=on 只证明 session
写保护,不证明读到了业务所需的新鲜数据。
34.7.2 用连接池、代理与应用控制点逐级减压
每层职责
application
request priority, deadline, idempotency, retry budget
PgBouncer
client waiting, server-pool concurrency, pool mode, reserve
HAProxy
role-aware backend eligibility, connection limits, queue, drain
PostgreSQL
hard backend limit, statement/lock/session guard, workload execution越靠上越理解业务价值,越靠下越能保护数据库硬边界。完整防线要组合,而不是希望 PgBouncer 一层解决所有过载。
先采配置事实
在变更前保存:
inventory revision and rendered config hash
service frontend/backend mapping
HAProxy health predicate and backend state
PgBouncer pool mode and per-database/user pool settings
client/server/waiting counts
PostgreSQL max/reserved/current connections
application instance and local-pool countsPigsty 可以通过 database/user 定义映射连接池参数;修改应回到声明式 inventory 和 受控部署流程。事故中对运行时做临时操作时,必须记录 drift,并在稳定后选择:
promote the change into config-as-code
or explicitly revert runtime state否则下次部署会“神秘地”覆盖救火配置。
减压顺序
1. application: reject/queue low priority and cap retry
2. scheduler: pause batch producers
3. pool: preserve OLTP/admin budget, constrain noisy class
4. proxy: drain ineligible or overloaded path when evidence supports it
5. database: exact cancel, timeout, role/session controls不要同时把 proxy backend 摘除和把应用全部转 direct;前者减少一条路径,后者可能在 另一条路径绕过所有 pool 限制。
故障切换期间的额外风险
切换会让旧 server connections 断开,新主同时接受大量重连。连接池要在新主 admission 前限速,HAProxy health 收敛与 Patroni role 收敛也要分别观测。恢复后逐级 prewarm, 不要让所有 pod 同时填满 pool。第 19、20、33 章分别给出连接路径、HA 和故障切换的 完整证据模型。
34.7.3 从面板判断范围,再用 SQL 与主机证据判型
面板适合回答“哪里、何时、范围多大”
Pigsty 的 PostgreSQL 监控可以把 cluster、instance、database、query、connection、 WAL、replication、checkpoint 和 host 指标放到同一时间线。事故开场先用面板定位:
single query / database / instance / whole cluster?
primary only or replicas too?
started at deploy, backup, checkpoint, traffic event, or failover?
connections, CPU, I/O, WAL and user latency which changed first?
is the signal rising, flat, oscillating, or recovering?面板是索引,不是最终证据。采样、聚合、label 和 exporter 失败都可能造成误解;告警 静默也可能是监控链坏了。
回到原生 SQL
流量型最小 SQL:
pg_stat_activity by app/state/wait
pg_blocking_pids root tree
pg_stat_statements workload deltas
pg_stat_database temp/deadlock/session deltas
pg_stat_io by backend/context保留型最小 SQL:
backend_xid/backend_xmin
pg_prepared_xacts
pg_replication_slots xmin/catalog_xmin/restart_lsn/status
pg_stat_archiver
replication receive/replay LSN
database/relation freeze ageSQL 再与主机证据互证:
filesystem mount and inode
device latency/queue/errors
memory pressure/swap/OOM
CPU run queue/steal/throttle
kernel and service events时间与身份要可关联
每份证据包含:
captured_at_utc: ...
observer: ...
cluster_instance_database: ...
query_or_command_version: ...
source_config_revision: ...
redaction: ...SQL 快照不要导出不必要的完整 query、口令、连接 URI 或业务数据。监控截图要保存 panel 时间窗、timezone、变量和 dashboard revision;否则复盘时无法重现。
平台动作的验收
Pigsty/HAProxy/PgBouncer state
says intended route/pool changed
PostgreSQL
proves actual role, session population, wait and retention state
host
proves resource pressure changed
synthetic/business probe
proves user-facing contract recovered四层有冲突时保留冲突,不要挑最漂亮的图作为结论。
上一节:保留型故障的安全路由 · 返回本章目录 · 下一节:实战:同一症状、两种成因 · 查看全书目录 · 查看索引中心