25.7 实战：实现并演练观察契约

本节把全章压成一条可重放的证据链：

bind ch24 observation contract
  -> validate signal and diagnostic-pack semantics
      -> compile 18 recording rules
          -> compile 13 alert rules
              -> preserve 7 accepted policies
                  -> quarantine 6 proposed policies
                      -> read live Pigsty/PostgreSQL baseline
                          -> verify target identity
                              -> run isolated VMAlert tests
                                  -> run offline Alertmanager routes
                                      -> test inhibition positive/negative
                                          -> reject 25 counterexamples
                                              -> publish 14-row coverage matrix
                                                  -> keep production gate pending

它刻意把两种动作分开：

online baseline
  L0, read-only, no mutation

isolated exercise
  L1-ephemeral, temporary loopback processes and files
  no online rule, no online alert, no real receiver

实验合同：

25.7.1 为延迟、错误、复制、备份和资源建立规则

文件布局

static/labs/ch25/
├── requirements.json
├── signal-contract.json
├── coverage-matrix.json
├── diagnostic-pack-contract.json
├── recording-rules.yml
├── alert-rules.yml
├── rule-tests.yml
├── alertmanager-sandbox.yml
├── route-tests.json
├── negative-cases.json
├── topology.mmd
├── lab-contract.md
├── capture.py
├── exercise.py
├── validate.py
├── review.py
├── task.sh
└── observability-run.json

职责：

文件	证明什么
requirements	target、risk、上游、hard rejection、gate
signal contract	source/type/label/reset/missing/cost
recording rules	SLI window 与诊断聚合
alert rules	accepted/proposed policy
rule tests	synthetic arithmetic、pending、firing、recovery
AM config	empty sink route 和 inhibition
route tests	八个 route、五个 inhibition 正反例
diagnostic pack	自动证据字段、limit、权限
coverage	已有信号、预期缺口、fallback
negative cases	validator 不是只会通过
capture	live L0 baseline
exercise	remote `/tmp` isolated test
review	file mode、secret、claim、count
public summary	allowlist 结果，不含 private evidence

上游 binding

第 24 章四份输入按 hash 绑定：

observation-contract.json
alert-candidates.json
slo-policy.json
governance-run.json

并要求：

run_id              34909737-527a-460c-927c-d9d71c93aa13
production_ch24_gate pending

如果上游改了 target、alert policy 或 missing semantics，本章 capture 不能继续假装基于旧合同。

记录规则：可用性

五个窗口：

rate5m
rate30m
rate1h
rate6h
rate3d

表达式：

sum by (service, operation_class, environment) (
  rate(pg36_shop_request_outcomes_total{
    eligible="true",
    outcome!="good"
  }[5m])
)
/
sum by (service, operation_class, environment) (
  rate(pg36_shop_request_outcomes_total{
    eligible="true"
  }[5m])
)

注意三个设计：

分子是 bad eligible event，不是 instance；
分母没有任意 clamp_min(...,1) 改变低流量比率；
缺失由独立 freshness/metamonitoring 处理，不补健康零。

instrumentation 必须预初始化 outcome category，或者用能区分“零 bad”与“bad series 未产生”的设计。

记录规则：延迟

1 -
sum by (service, operation_class, environment) (
  rate(pg36_shop_request_duration_seconds_bucket{
    eligible="true",
    le="0.25"
  }[5m])
)
/
sum by (service, operation_class, environment) (
  rate(pg36_shop_request_duration_seconds_count{
    eligible="true"
  }[5m])
)

产生 5m/1h bad ratio。

规则可计算，但 alert governance 尚未接受：

route: test
severity: candidate
governance_status: proposed-not-accepted

记录规则：新鲜度

sum by (service, read_path, environment) (
  rate(pg36_shop_commit_visibility_probes_total{
    eligible="true",
    within_bound="false"
  }[5m])
)
/
sum by (service, read_path, environment) (
  rate(pg36_shop_commit_visibility_probes_total{
    eligible="true"
  }[5m])
)

它不读取 pg_lag，因为用户 SLI 必须携带 commit token。

记录规则：PostgreSQL 诊断

pg36:replica_replay_distance_bytes:max
pg36:archive_failures:increase15m
pg36:longest_transaction_seconds:max
pg36:table_freeze_age:max
pg36:dead_tuples:sum
pg36:exporter_unavailable:max

这些是原因/容量输入，不自动 page。

记录规则：metamonitoring

pg36:vmalert_rule_errors:sum
pg36:vmalert_missed_iterations:increase15m
pg36:notification_failures:increase15m

一个重要限制：notification failure 为零仍不能证明 receiver receipt。还需要外部 canary。

accepted 七条

validator 逐字段与第 24 章比对：

alert	route	`for`
availability fast	page	2m
availability slow	page	5m
availability budget	ticket	15m
freshness fast	page	2m
correctness mismatch	page	0m
capacity horizon	ticket	1h
monitoring path broken	page	5m

比较：

-route； -severity； -objective； -owner； -runbook； -first safe action； -governance status； -for； -multiwindow token。

任意漂移使 lint 失败。

proposed 六条

alert	用途	为什么不直接上线
latency fast burn	用户延迟	ch24 尚未接受 page policy
restore evidence stale	recovery control	route/severity 待接受
archive failure active	recovery risk	threshold/pgBackRest 联动待评审
transaction age	maintenance/capacity	workload-specific policy
freeze age	wraparound horizon	应从配置/rate 生成
SLI missing	observation path	expected traffic/fallback 待实现

统一：

route: test
severity: candidate
governance_status: proposed-not-accepted

validator 会拒绝 candidate 进入 page/ticket sink。

archive candidate 不比较历史总数

pg36:archive_failures:increase15m > 0
and on (cls, ins, ip)
(time() - pg_archiver_finish_time) > 900

同时要求：

-15 分钟出现新失败； -成功推进已经停滞。

仍需 runbook 复核 pgBackRest 和 restore evidence。

transaction candidate

pg36:longest_transaction_seconds:max > 900

只走 test，因为：

-batch 可能合法； -idle in transaction 与 active 不同； -cancel 权限与 unknown outcome； -阈值依赖 workload； -user impact 可能不存在。

first action 是找 owner/queryid 与安全性，不是 terminate。

freeze candidate

pg36:table_freeze_age:max > 1000000000

固定值仅用于 synthetic test。生产应使用：

current age
current autovacuum_freeze_max_age
table override
consumption rate
vacuum throughput
blocker
safety margin

capacity accepted 规则仍需 reviewed input

pg36_shop_capacity_horizon_days < 14
and on (service, environment)
pg36_shop_capacity_forecast_reviewed == 1

accepted 是告警合同，未表示 forecast exporter 已实现。coverage 仍标记应用层缺口。

rules 与 groups

三组 recording 和四组 alert 分开。原因是 VMAlert 的 recording result remote write 是异步的；同 group 依赖上一条结果可能读不到本轮。

25 个反例之一会把 alert 塞回 recording group，确认 validator 拒绝：

recording group contains an alert and creates rule chaining

当前 live baseline

capture.py 只读：

VictoriaMetrics   health/API/query
VictoriaLogs      health/version
VictoriaTraces    health/version
VMAlert           rules/alerts/self metrics
Alertmanager      health/alert count/self metrics
pg_exporter       raw HELP/name/label inventory
PgBouncer exporter response
Patroni           selected identity/state
PostgreSQL        bounded read-only SQL

不会读取 Alertmanager config，因为可能有 receiver secret；只保存 count 与 configuration_exported=false。

PostgreSQL target identity

工作站配置：

10.10.10.10 -> local port 2222
10.10.10.11 -> local port 2200
...

现场发现多个转发落到元节点。正式路径改为：

ssh ProxyJump meta
  target HostName=10.10.10.11 inside sandbox

并验证：

hostname            pg-test-1
cluster_name        pg-test
Patroni             pg-test/pg-test-1
pg_is_in_recovery   false
replication rows    2

这一检查会拒绝“查询成功但目标错误”。

实际 PG observation settings

正式快照记录：

track_activities              on
track_counts                  on
track_io_timing               on
track_wal_io_timing           off
stats_fetch_consistency       cache

pg_stat_statements.track      all
track_planning                off
track_utility                 off

logging_collector             on
log_destination               csvlog
log_file_mode                 0640

auto_explain.log_min_duration 1000ms
auto_explain.log_analyze      on
auto_explain.log_timing       on
auto_explain.sample_rate      1

两个待评审：

auto_explain overhead reviewed  false
log group access reviewed       false

实验不修改配置。

25.7.2 注入可控症状，验证触发、路由、抑制和恢复

只做静态 lint

static/labs/ch25/task.sh lint

输出：

recording_rules=18
alert_rules=13
accepted_alerts=7
proposed_alerts=6
counterexamples=25-rejected
live_deployment=false
production_ch25_gate=pending

不访问远端。

正式完整运行

先创建一个新的私密路径；task.sh 也会拒绝覆盖非空目录：

export PG36_EVIDENCE_DIR=/absolute/private/new-empty/ch25-run
static/labs/ch25/task.sh all

动作：

capture
  L0 live read-only

exercise
  L1 remote temp / loopback only

verify
  positive + negative + source hash + live evidence

review
  private mode + secret scan + claim boundary

capture 的 SQL 安全边界

statement_timeout=5s
lock_timeout=500ms
default_transaction_read_only=on
BEGIN READ ONLY

query allowlist 不含：

query text
bind
client address
EXPLAIN ANALYZE
reset
vacuum/checkpoint
cancel/terminate
DDL/DML

即便如此，pg_locks 会看到采集查询自身的 relation lock；这是 observer effect，证据不会假装完全无影响。

isolated workspace

exercise.py：

在元节点执行 mktemp -d /tmp/pg36-ch25.XXXXXXXX；
验证路径符合严格 regex；
只上传四个无 secret 文件；
执行测试；
rm -rf 只允许匹配该 prefix；
断言目录已经不存在；
输出 remote_cleanup=ok。

如果 temp path 不匹配，宁可失败也不清理未知目录。

规则语法

vmalert -rule=recording-rules.yml
        -rule=alert-rules.yml
        -dryRun

证明 parser/规则结构可接受，不执行 live query。

合成时序

vmalert-tool：

isolated VictoriaMetrics
simulated periodic ingestion
recording/alert evaluation
expected sample/alert comparison

availability recording 输入：

good +100/min
bad  +2/min

验证结果：

$$ \frac{2}{102}=0.019607843… $$

fast pending/firing/recovery

输入：

1h bad ratio  0.02 for 4 samples, then 0
5m bad ratio  0.02 for 4 samples, then 0
evaluation    1m
for           2m

期望：

1m  no firing alert
3m  exact alert labels/annotations
7m  no alert

这验证 expression、for、label identity 和 recovery。

其他合成用例

availability slow burn
availability budget ticket
freshness fast burn
correctness immediate
reviewed capacity ticket
monitoring path page
latency proposed route
expected traffic but SLI missing

correctness for=0m，capacity 需要 reviewed gauge，missing 不补零。

Alertmanager 配置检查

amtool check-config

发现：

global config
route
2 inhibition rules
7 receivers
0 templates

所有 receiver 只有 name，没有 integration。

八个 route

用例	期望 sink
availability page	`pg36-shop-page-sink`
correctness page	`pg36-shop-page-sink`
service ticket	`pg36-shop-ticket-sink`
platform ticket	`pg36-platform-ticket-sink`
metamonitoring page	`pg36-platform-page-sink`
diagnostic	`pg36-null-sink`
proposed	`pg36-proposed-rule-sink`
unknown	`pg36-default-sink`

使用：

amtool config routes test --config.file=...
  --verify.receivers=<expected>
  <labels...>

离线解析，不连接 live Alertmanager。

五个 inhibition

source → target	结果
fast → same service/objective slow	inhibit
meta → derived missing	inhibit
meta → independent user symptom	do not
meta → correctness	do not
fast → other service slow	do not

validator 同时解析 config 和测试表；添加 broad correctness inhibition 会被反例拒绝。

25 个对抗性变体

类别：

scope
  production data/approval promoted

online mutation
  live deploy/notification/Alertmanager

semantics
  missing healthy, timing disabled = zero

privacy
  tenant label, query text, public evidence

claims
  app metric/rules/pager claimed live

query safety
  EXPLAIN ANALYZE

alert governance
  accepted route/for drift, candidate page

time
  slow rule loses second window

component semantics
  archive historical counter

engine
  alert chained in recording group

routing
  real webhook, correctness inhibition, real receiver name

validator 不是检查“文件存在”，而是逐个 mutation 后要求出现预期 rejection。

正式运行结果

run_id      b876731e-5741-40e3-a03e-17cf7d20881b
captured    2026-07-29T22:48:21Z
target      pg36-l2-vagrant/pg-test
live risk   L0
exercise    L1-ephemeral

组件：

Pigsty            v4.4.0
PostgreSQL        18.4
pg_exporter       v1.4.0
VictoriaMetrics   v1.148.0
VictoriaLogs      v1.52.0
VictoriaTraces    v0.9.4
Alertmanager      0.33.1

在线：

VM series                  44,842
label-value pairs          387,724
VMAlert groups             17
live alert/recording       50 / 698
live rule errors           0
live current alerts        0
application SLI series     0

PostgreSQL：

primary                 pg-test-1
async replicas          2
observed WAL gap        0 / 0 bytes
pg_stat_statements rows 194
pgss calls              122,303
query text exported     false
archive failed_count    21
last success after fail true

演练：

18 recording rules       pass
13 alert rules           pass
8 routes                 pass
5 inhibition cases       pass
25 counterexamples       rejected
remote cleanup           pass
secret scan              pass

结论：

live chapter deployment  false
real receiver            false
production_ch25_gate     pending

evidence

private bundle：

observability-evidence.json
isolated-exercise.txt
validation-report.json
negative-report.json
review.txt

全部：

directory 0700
files     0600

公共摘要只包含 allowlist，不包含 source body、SQL text、log body 或 receiver config。

重验

export PG36_EVIDENCE_DIR=/absolute/private/existing/ch25-run
static/labs/ch25/task.sh verify
static/labs/ch25/task.sh review

如果 source file 在 capture 后改变，hash 校验失败。必须新跑，而不是手工改 report。

没有 reset action

本章：

-在线没有 mutation； -remote temp 自动清理； -isolated process 退出； -没有 database fixture； -没有 live rule。

所以没有数据库 reset。不要添加一个“为了形式完整”的 destructive reset。

25.7.3 输出告警覆盖表、盲区与 ch31 的诊断入口

14 行覆盖矩阵

coverage-matrix.json 包含：

5 user/control signals
6 PostgreSQL/platform observation areas
logs
traces
routing

每行：

question
source
live expected/status
rule status
fallback
owner
blind spot

五个应用缺口

coverage	live	fallback
availability event	absent	independent place-order probe
latency histogram	absent	end-to-end duration probe
commit freshness	absent	unique-token probe
correctness reconciliation	absent	signed reconciliation
restore evidence age	absent	immutable restore manifest

它们是 expected-gap，因为教学沙箱没有对应应用。

不得由：

pg_up
HAProxy up
pg_lag
backup success
zero DB errors

代填。

已有 PostgreSQL/platform 覆盖

exporter reachability
activity/transaction/wait/lock
I/O/WAL/checkpoint/archive/replication
maintenance/freeze/object
pg_stat_statements aggregate/reset
VMAlert/Alertmanager self monitoring
VictoriaLogs endpoint
VictoriaTraces endpoint

“已有”仍带盲区：

-one sample misses transient wait； -PG I/O 不区分 OS cache/device； -dead tuple 不是 exact bloat； -queryid collision； -endpoint 不证明 body/parse； -backend 不证明 application spans； -notification failure 0 不证明 receipt。

配置观察到的待评审项

`auto_explain`

log_analyze   on
log_timing    on
sample_rate   1
threshold     1s
nested        on
parameter max -1

生产前：

-代表性 load test； -short statement overhead； -log volume； -parameter leakage； -log_timing=off 方案； -sampling； -rollback； -owner approval。

本章不直接说“必须关闭”，也不说“当前安全”。

log `0640`

可能为 Vector collector group 提供 read。生产前：

-actual owner/group； -group membership； -directory mode； -rotation； -central storage ACL； -retention； -redaction； -access audit。

alert coverage 不等于 implementation

七条 accepted alert 有规则，但 app source 不存在：

policy coverage   yes
synthetic test    yes
live data         no
live deploy       no
real route        no

因此不能说“availability monitoring complete”。

proposed queue

需要治理拍板：

latency page route/severity；
restore evidence stale 的 change gate；
archive risk 的 threshold 与 pgBackRest 证据；
long transaction workload class；
freeze horizon based on config/rate；
expected traffic/missing source；
real receipt canary。

每项完成后：

-更新 ch24 governance； -hash binding 失效； -重跑 ch25； -canary deploy； -验证 route/receipt； -保留 rollback。

生产上线前的门禁

service instrumentation
  event schema/version/cardinality

source validation
  producer/zero/missing/reset

rule validation
  synthetic + shadow/live query

governance
  owner/runbook/first action/severity

routing
  real receiver secret outside Git

delivery
  receipt canary

cost
  exporter/query/storage/log overhead

privacy
  labels/body/parameters/retention/access

deployment
  staged canary, rollback, change event

operations
  drill and evidence

完成之前：

production_ch25_gate=pending

ch31 诊断入口

本章输出按 symptom 路由给第 31 章：

慢查询

user latency windows
entry/pool wait
activity wait
pg_stat_statements reset + top queryid
I/O/temp/WAL
release/plan event
host evidence

锁与死锁

user outcome
current blocker graph
lock wait/deadlock log aggregate
transaction age/xmin
application owner
safe cancellation boundary

连接与池

edge queue
HAProxy/PgBouncer client/server/wait
PG active/idle/idle-in-xact
connection config change
max connection headroom

复制与新鲜度

commit token probe
read path
Patroni role/timeline
sender/receiver position
slot/WAL retention
route change

磁盘与容量

filesystem horizon
PG I/O bytes/time
device latency/queue
WAL/archive
object growth
rewrite/backup headroom

vacuum 与冻结

old xact/xmin
table estimates
progress
freeze age/config/rate
slot/feedback
host I/O

诊断入口的统一字段

run id
captured at
service/environment
target identity
symptom windows/count
topology
changes
PostgreSQL reset and aggregates
platform freshness
known gaps
hypotheses/falsifiers
risk/authority

第 31 章不需要从“打开所有 dashboard”重新开始。

完成标准

本章不是以“31 条规则存在”完成，而是：

semantics are explicit
accepted and proposed are separated
live and isolated are separated
missing is not healthy
rules are tested
routes are tested without delivery
inhibition has negative tests
native PG evidence is bounded
privacy and cost are explicit
blind spots are published
production gate remains honest

本节验收

你应当能复述：

18 条 recording rule 分哪三组；
13 条 alert 中哪七条 accepted、哪六条 proposed；
archive rule 为什么同时看增量与 last success；
target identity 为什么需要 ProxyJump 后三源校验；
capture 与 exercise 的风险/权限边界；
synthetic time series 如何验证 pending/firing/recovery；
route test 为什么使用空 receiver；
inhibition 为什么必须有 must-not-inhibit；
25 个反例覆盖哪些失败模式；
private evidence 与 public summary 如何分离；
五个 application gap 为什么不能用 PG component metric 替代；
auto_explain 与 0640 为什么是待评审而非自动修复；
生产门禁还缺哪些步骤；
第 31 章如何使用本章诊断包。

上一节：从告警到诊断包 · 返回本章目录 · 下一章：胸有成竹：容量规划与压测基线 · 查看全书目录 · 查看索引中心

最后更新于 2026-07-30

25.6 从告警到诊断包