rule_files: - recording-rules.yml - alert-rules.yml evaluation_interval: 1m group_eval_order: - pg36-shop-sli-recording - pg36-postgresql-diagnostic-recording - pg36-observation-meta-recording - pg36-shop-slo-alerts - pg36-shop-control-alerts - pg36-postgresql-proposed-alerts - pg36-observation-path-alerts tests: - name: availability recording rule preserves event ratio interval: 1m input_series: - series: 'pg36_shop_request_outcomes_total{service="pg36_shop",operation_class="place-order",environment="isolated-lab",eligible="true",outcome="good"}' values: "100+100x10" - series: 'pg36_shop_request_outcomes_total{service="pg36_shop",operation_class="place-order",environment="isolated-lab",eligible="true",outcome="failed"}' values: "0+2x10" metricsql_expr_test: - expr: pg36_shop:sli_availability_bad_ratio:rate5m eval_time: 10m exp_samples: - labels: 'pg36_shop:sli_availability_bad_ratio:rate5m{environment="isolated-lab",operation_class="place-order",service="pg36_shop"}' value: 0.01960784313725 - name: availability fast burn pending firing and recovery interval: 1m input_series: - series: 'pg36_shop:sli_availability_bad_ratio:rate1h{service="pg36_shop",operation_class="place-order",environment="isolated-lab"}' values: "0.02x3 0x5" - series: 'pg36_shop:sli_availability_bad_ratio:rate5m{service="pg36_shop",operation_class="place-order",environment="isolated-lab"}' values: "0.02x3 0x5" alert_rule_test: - eval_time: 1m groupname: pg36-shop-slo-alerts alertname: PG36ShopAvailabilityFastBurn exp_alerts: [] - eval_time: 3m groupname: pg36-shop-slo-alerts alertname: PG36ShopAvailabilityFastBurn exp_alerts: - exp_labels: service: pg36_shop operation_class: place-order environment: isolated-lab class: symptom route: page severity: SEV-1 objective_id: SLO-AVAILABILITY owner_function: service runbook_id: RB-USER-SYMPTOM governance_status: accepted-ch24 observation_dependency: "false" exp_annotations: summary: Availability error budget is burning fast user_impact: Eligible order attempts are failing or have unreconciled outcomes first_safe_action: freeze the latest risky release and reconcile unknown order outcomes before retrying writes verification: Both burn windows recover and sampled idempotency tokens reconcile dashboard: dashboard://pg36-shop-slo - eval_time: 7m groupname: pg36-shop-slo-alerts alertname: PG36ShopAvailabilityFastBurn exp_alerts: [] - name: availability slow burn interval: 1m input_series: - series: 'pg36_shop:sli_availability_bad_ratio:rate6h{service="pg36_shop",operation_class="place-order",environment="isolated-lab"}' values: "0.01x8" - series: 'pg36_shop:sli_availability_bad_ratio:rate30m{service="pg36_shop",operation_class="place-order",environment="isolated-lab"}' values: "0.01x8" alert_rule_test: - eval_time: 6m groupname: pg36-shop-slo-alerts alertname: PG36ShopAvailabilitySlowBurn exp_alerts: - exp_labels: service: pg36_shop operation_class: place-order environment: isolated-lab class: symptom route: page severity: SEV-2 objective_id: SLO-AVAILABILITY owner_function: service runbook_id: RB-USER-SYMPTOM governance_status: accepted-ch24 observation_dependency: "false" exp_annotations: summary: Availability error budget is burning persistently user_impact: Sustained failed or unreconciled order attempts threaten the objective first_safe_action: stop concurrent risky changes and segment failures by operation and release verification: Both burn windows recover and the responsible failure class is bounded dashboard: dashboard://pg36-shop-slo - name: availability budget ticket interval: 1m input_series: - series: 'pg36_shop:sli_availability_bad_ratio:rate3d{service="pg36_shop",operation_class="place-order",environment="isolated-lab"}' values: "0.002x20" - series: 'pg36_shop:sli_availability_bad_ratio:rate6h{service="pg36_shop",operation_class="place-order",environment="isolated-lab"}' values: "0.002x20" alert_rule_test: - eval_time: 16m groupname: pg36-shop-slo-alerts alertname: PG36ShopAvailabilityBudgetTicket exp_alerts: - exp_labels: service: pg36_shop operation_class: place-order environment: isolated-lab class: symptom route: ticket severity: ticket objective_id: SLO-AVAILABILITY owner_function: service runbook_id: RB-BUDGET-REVIEW governance_status: accepted-ch24 observation_dependency: "false" exp_annotations: summary: Availability budget requires an owned review user_impact: The sustained rate would exhaust the objective budget first_safe_action: open an owned budget review and identify the largest eligible failure class verification: The ticket contains event counts windows release markers and a due date dashboard: dashboard://pg36-shop-slo - name: commit correlated freshness fast burn interval: 1m input_series: - series: 'pg36_shop:sli_freshness_bad_ratio:rate1h{service="pg36_shop",read_path="replica",environment="isolated-lab"}' values: "0.2x5" - series: 'pg36_shop:sli_freshness_bad_ratio:rate5m{service="pg36_shop",read_path="replica",environment="isolated-lab"}' values: "0.2x5" alert_rule_test: - eval_time: 3m groupname: pg36-shop-slo-alerts alertname: PG36ShopFreshnessFastBurn exp_alerts: - exp_labels: service: pg36_shop read_path: replica environment: isolated-lab class: symptom route: page severity: SEV-2 objective_id: SLO-FRESHNESS owner_function: service runbook_id: RB-READ-FRESHNESS governance_status: accepted-ch24 observation_dependency: "false" exp_annotations: summary: Commit-correlated reads violate the freshness bound user_impact: Routed reads are missing known committed tokens beyond five seconds first_safe_action: route the affected read-after-write journey to the primary path while preserving probe tokens verification: New commit tokens become visible within five seconds on the declared path dashboard: dashboard://pg36-shop-freshness - name: correctness is immediate and cannot spend error budget interval: 1m input_series: - series: 'pg36_shop_reconciliation_mismatches{service="pg36_shop",invariant="order-uniqueness",environment="isolated-lab"}' values: "1x2" alert_rule_test: - eval_time: 0m groupname: pg36-shop-control-alerts alertname: PG36ShopCorrectnessMismatch exp_alerts: - exp_labels: service: pg36_shop invariant: order-uniqueness environment: isolated-lab class: integrity route: page severity: SEV-1 objective_id: CTRL-CORRECTNESS owner_function: data runbook_id: RB-CORRECTNESS governance_status: accepted-ch24 observation_dependency: "false" exp_annotations: summary: Domain reconciliation found unexplained mismatches user_impact: Acknowledged order state may be duplicated invalid cross-tenant or missing first_safe_action: freeze affected writes and preserve the reconciliation boundary before repair verification: An independently reviewed reconciliation returns zero unexplained mismatches dashboard: dashboard://pg36-shop-correctness - name: reviewed capacity forecast creates a ticket not a page interval: 1m input_series: - series: 'pg36_shop_capacity_horizon_days{service="pg36_shop",environment="isolated-lab"}' values: "10x65" - series: 'pg36_shop_capacity_forecast_reviewed{service="pg36_shop",environment="isolated-lab"}' values: "1x65" alert_rule_test: - eval_time: 61m groupname: pg36-shop-control-alerts alertname: PG36ShopCapacityHorizon exp_alerts: - exp_labels: service: pg36_shop environment: isolated-lab class: capacity route: ticket severity: ticket owner_function: platform runbook_id: RB-CAPACITY governance_status: accepted-ch24 observation_dependency: "false" exp_annotations: summary: Reviewed capacity horizon is below fourteen days user_impact: No current impact is asserted; avoidable future saturation risk exists first_safe_action: validate demand, headroom, retention, and query mix before changing capacity verification: An owned plan restores the reviewed headroom horizon dashboard: dashboard://pg36-shop-capacity - name: monitoring path failure pages the platform role interval: 1m input_series: - series: 'pg36_observation_path_broken{service="pg36_shop",environment="isolated-lab"}' values: "1x7" alert_rule_test: - eval_time: 6m groupname: pg36-observation-path-alerts alertname: PG36MonitoringPathBroken exp_alerts: - exp_labels: service: pg36_shop environment: isolated-lab class: metamonitoring route: page severity: SEV-2 owner_function: platform runbook_id: RB-METAMONITORING governance_status: accepted-ch24 observation_dependency: "false" exp_annotations: summary: Observation or notification path cannot prove it is working user_impact: Service health is unknown because the observation chain is broken first_safe_action: establish an independent blackbox view before changing the monitored database verification: A canary traverses probe ingestion evaluation routing and receipt dashboard: dashboard://pigsty-observability - name: latency rule remains a proposed test route interval: 1m input_series: - series: 'pg36_shop:sli_latency_bad_ratio:rate1h{service="pg36_shop",operation_class="place-order",environment="isolated-lab"}' values: "0.2x5" - series: 'pg36_shop:sli_latency_bad_ratio:rate5m{service="pg36_shop",operation_class="place-order",environment="isolated-lab"}' values: "0.2x5" alert_rule_test: - eval_time: 3m groupname: pg36-shop-slo-alerts alertname: PG36ShopLatencyFastBurn exp_alerts: - exp_labels: service: pg36_shop operation_class: place-order environment: isolated-lab class: symptom route: test severity: candidate objective_id: SLO-LATENCY owner_function: service runbook_id: RB-USER-LATENCY-CANDIDATE governance_status: proposed-not-accepted observation_dependency: "false" exp_annotations: summary: Candidate latency burn rule fired in the isolated lab user_impact: Eligible order attempts may exceed the declared latency bound first_safe_action: Correlate the user histogram with entry queueing and database waits without changing the database verification: Governance accepts the policy and both tested windows recover dashboard: dashboard://pg36-shop-slo - name: missing SLI is unknown rather than healthy interval: 1m input_series: - series: 'pg36_expected_service_traffic{service="pg36_shop",environment="isolated-lab"}' values: "1x7" alert_rule_test: - eval_time: 6m groupname: pg36-observation-path-alerts alertname: PG36ShopSLIMissing exp_alerts: - exp_labels: service: pg36_shop environment: isolated-lab class: derived-missing route: test severity: candidate owner_function: platform runbook_id: RB-SLI-MISSING-CANDIDATE governance_status: proposed-not-accepted observation_dependency: "true" exp_annotations: summary: Expected service traffic has no fresh SLI sample user_impact: Service health is unknown; this is not a healthy zero first_safe_action: Check the independent probe and ingestion path before changing the service verification: Fresh samples and the independent probe agree dashboard: dashboard://pigsty-observability