跳至内容

34.7 平台级流量控制与证据

Pigsty 把 PostgreSQL、Patroni、HAProxy、PgBouncer、监控和配置管理组合成平台。它提供 多个可以减压或隔离的控制点,但不会替操作者判断一致性语义,也不会把一条面板曲线 自动变成根因。

34.7.1 从服务端点隔离批处理和只读流量

端口背后是服务契约

Pigsty 的默认服务意图通常是:

服务常见端口默认目标适用工作
primary5433当前主库的 PgBouncer短 OLTP 读写
replica5434可读节点的 PgBouncer可容忍副本语义的读
default5436当前主库 PostgreSQL 直连管理、迁移、session-sensitive
offline5438offline/replica PostgreSQL 直连受控 OLAP/ETL

这些是参考配置,不是 PostgreSQL 固有端口;必须以当前 inventory 与生成配置为准。 HAProxy 通常用 Patroni role endpoint 判后端资格,PgBouncer 再把大量 client connection 映射为受控 server connection。

正确隔离:

OLTP write -> primary pooled service
read-with-staleness-contract -> replica pooled service
long report/ETL -> offline service + separate budget
DDL/admin/session semantics -> controlled direct service

不正确隔离:

all SELECT -> replica, regardless of read-your-write
all long queries -> offline, regardless of shared storage/CPU
database slow -> send everything to every replica
primary service full -> bypass PgBouncer through direct service

direct service 是为需要 session/管理语义的受控客户端准备的路径,不是池满时的逃生 后门。若应用能无治理地从 5433 切到 5436,连接预算就失去意义。

路由变化也要验收数据语义

切到 replica/offline 后检查:

SELECT pg_is_in_recovery(),
       current_setting('transaction_read_only'),
       pg_last_wal_receive_lsn(),
       pg_last_wal_replay_lsn(),
       pg_last_xact_replay_timestamp();

还要用业务 token 验证 staleness/可见性。transaction_read_only=on 只证明 session 写保护,不证明读到了业务所需的新鲜数据。

34.7.2 用连接池、代理与应用控制点逐级减压

每层职责

application
  request priority, deadline, idempotency, retry budget

PgBouncer
  client waiting, server-pool concurrency, pool mode, reserve

HAProxy
  role-aware backend eligibility, connection limits, queue, drain

PostgreSQL
  hard backend limit, statement/lock/session guard, workload execution

越靠上越理解业务价值,越靠下越能保护数据库硬边界。完整防线要组合,而不是希望 PgBouncer 一层解决所有过载。

先采配置事实

在变更前保存:

inventory revision and rendered config hash
service frontend/backend mapping
HAProxy health predicate and backend state
PgBouncer pool mode and per-database/user pool settings
client/server/waiting counts
PostgreSQL max/reserved/current connections
application instance and local-pool counts

Pigsty 可以通过 database/user 定义映射连接池参数;修改应回到声明式 inventory 和 受控部署流程。事故中对运行时做临时操作时,必须记录 drift,并在稳定后选择:

promote the change into config-as-code
or explicitly revert runtime state

否则下次部署会“神秘地”覆盖救火配置。

减压顺序

1. application: reject/queue low priority and cap retry
2. scheduler: pause batch producers
3. pool: preserve OLTP/admin budget, constrain noisy class
4. proxy: drain ineligible or overloaded path when evidence supports it
5. database: exact cancel, timeout, role/session controls

不要同时把 proxy backend 摘除和把应用全部转 direct;前者减少一条路径,后者可能在 另一条路径绕过所有 pool 限制。

故障切换期间的额外风险

切换会让旧 server connections 断开,新主同时接受大量重连。连接池要在新主 admission 前限速,HAProxy health 收敛与 Patroni role 收敛也要分别观测。恢复后逐级 prewarm, 不要让所有 pod 同时填满 pool。第 19、20、33 章分别给出连接路径、HA 和故障切换的 完整证据模型。

34.7.3 从面板判断范围,再用 SQL 与主机证据判型

面板适合回答“哪里、何时、范围多大”

Pigsty 的 PostgreSQL 监控可以把 cluster、instance、database、query、connection、 WAL、replication、checkpoint 和 host 指标放到同一时间线。事故开场先用面板定位:

single query / database / instance / whole cluster?
primary only or replicas too?
started at deploy, backup, checkpoint, traffic event, or failover?
connections, CPU, I/O, WAL and user latency which changed first?
is the signal rising, flat, oscillating, or recovering?

面板是索引,不是最终证据。采样、聚合、label 和 exporter 失败都可能造成误解;告警 静默也可能是监控链坏了。

回到原生 SQL

流量型最小 SQL:

pg_stat_activity by app/state/wait
pg_blocking_pids root tree
pg_stat_statements workload deltas
pg_stat_database temp/deadlock/session deltas
pg_stat_io by backend/context

保留型最小 SQL:

backend_xid/backend_xmin
pg_prepared_xacts
pg_replication_slots xmin/catalog_xmin/restart_lsn/status
pg_stat_archiver
replication receive/replay LSN
database/relation freeze age

SQL 再与主机证据互证:

filesystem mount and inode
device latency/queue/errors
memory pressure/swap/OOM
CPU run queue/steal/throttle
kernel and service events

时间与身份要可关联

每份证据包含:

captured_at_utc: ...
observer: ...
cluster_instance_database: ...
query_or_command_version: ...
source_config_revision: ...
redaction: ...

SQL 快照不要导出不必要的完整 query、口令、连接 URI 或业务数据。监控截图要保存 panel 时间窗、timezone、变量和 dashboard revision;否则复盘时无法重现。

平台动作的验收

Pigsty/HAProxy/PgBouncer state
  says intended route/pool changed

PostgreSQL
  proves actual role, session population, wait and retention state

host
  proves resource pressure changed

synthetic/business probe
  proves user-facing contract recovered

四层有冲突时保留冲突,不要挑最漂亮的图作为结论。


上一节:保留型故障的安全路由 · 返回本章目录 · 下一节:实战:同一症状、两种成因 · 查看全书目录 · 查看索引中心

最后更新于