Database Failover Game Day Runbook【免费下载链接】claude-skills67 Specialized Skills for Full-Stack Developers. Transform Claude Code into your expert pair programmer.项目地址: https://gitcode.com/GitHub_Trending/claud/claude-skillsDate: January 15, 2025Duration: 2 hoursEnvironment: StagingPre-Game Checklist (T-30 min)Verify all participants joined war roomConfirm monitoring dashboards accessibleTest rollback procedures workAnnounce game day start in #engineeringVerify staging environment healthySet up screen recording for timelinePrepare incident timeline spreadsheetTimeline10:00 - Introduction (10 min)Facilitator explains objectivesReview scenarios and success criteriaConfirm roles and communication channelsRemind everyone: this is a learning exercise10:10 - Scenario 1: Primary DB Failure (30 min)T0 (10:10)- Inject failureaws rds reboot-db-instance \ --db-instance-identifier staging-primary \ --force-failoverExpected Timeline:T0: Reboot initiatedT30s: Primary becomes unavailableT60s: DNS updated to standbyT90s: Application reconnectsT120s: Full recoveryObserver Tasks:Record exact time of failure injectionMonitor application error logsTrack alert notificationsDocument team response actionsScreenshot dashboard statesQuestions to Answer:How long until first alert?Did application auto-reconnect?Were customers impacted?What manual interventions needed?10:40 - Debrief Scenario 1 (10 min)What went well?What could improve?Any surprises?Action items identified10:50 - Scenario 2: Network Partition (20 min)T0 (10:50)- Inject failure# Block database security group ingress aws ec2 revoke-security-group-ingress \ --group-id sg-xxxxx \ --protocol tcp \ --port 5432 \ --cidr 10.0.0.0/16Expected Behavior:Connection timeouts occurCircuit breaker opensRead-only mode activatesClear error messages shownObserver Tasks:Monitor circuit breaker stateVerify read-replica failoverCheck user-facing error messagesTrack degraded service duration11:10 - Debrief Scenario 2 (10 min)11:20 - Scenario 3: Surprise! (20 min)Facilitator Note: Dont announce this scenario details beforehand. Test true incident response capability.Hidden Scenario: Combination failureDatabase connection pool leakSimultaneous cache invalidation# Connection leak simulator import psycopg2 connections [] for i in range(100): conn psycopg2.connect(DATABASE_URL) connections.append(conn) # Intentionally dont closeObserver Tasks:How long to identify root cause?Communication effectivenessCross-team coordinationEscalation decisions11:40 - Final Debrief Wrap-up (20 min)Debrief Questions:What worked well?What didnt work?What surprised us?What are our top 3 action items?When should we run this again?Post-Game ChecklistRestore all services to normal stateVerify no lingering issuesCollect all observer notesExport metrics and dashboardsSchedule post-mortem meetingSend thank-you to participantsCreate action item ticketsUpdate runbooks based on learnings这份 Runbook 的编排逻辑非常讲究 - **场景之间插入 Debrief**每个场景执行后立即用 10 分钟做热复盘趁记忆新鲜捕捉 What went well / What could improve / Any surprises / Action items而不是等全部演练结束再一次性回忆。 - **Surprise 场景对参与者保密**第三个场景惊喜测试的细节不提前宣布专门检验团队的真实响应能力而非照着剧本表演。这种设计在原始文档中明确标注为 Facilitator Note。 - **预期时间线预先建立**T0 / T30s / T60s / T90s / T120s 的预期恢复节奏让观察者能用实际 vs 预期的偏差快速定位系统短板。 - **赛后恢复清单**Post-Game Checklist 中的Restore all services to normal state是演练伦理的底线——演练必须在可控范围内收场不能给业务留下持续的副作用。 故障注入命令本身也值得注意场景一用 aws rds reboot-db-instance --force-failover 模拟主库故障转移场景二用 aws ec2 revoke-security-group-ingress 阻断数据库安全组的网络入口模拟网络分区。这两个命令都是真实可执行的生产级操作直接对应了 [infrastructure-chaos.md](https://link.gitcode.com/i/41bfc2b404aaeca6cc691499c56ac0c0) 中的 AWS 故障模拟体系——该文档用 boto3 封装了 simulate_az_failure()按可用区终止 ASG 实例并挂起 AZRebalance 进程与 drain_az_from_load_balancer()将目标可用区从负载均衡摘除等更精细的注入手段。 --- ## 四、观测与量化Observation 与 Metrics 模板 Game Day 区别于随便搞坏再修复的关键在于**全程可量化的观测**。原始文档提供了两个 Python dataclass 作为观测与指标的标准数据结构既可用于演练现场实时记录也可沉淀为自动化工具的基础。 python from dataclasses import dataclass, field from datetime import datetime from typing import List dataclass class GameDayObservation: timestamp: datetime observer: str scenario: str observation: str category: str # technical, process, communication, surprise severity: str # info, concern, critical photo_url: str dataclass class GameDayMetrics: scenario_name: str start_time: datetime end_time: datetime # Technical metrics time_to_detect_seconds: float time_to_respond_seconds: float time_to_recover_seconds: float error_rate_peak: float alerts_fired: List[str] field(default_factorylist) alerts_missed: List[str] field(default_factorylist) # Team metrics responders_involved: int escalations_needed: int communication_gaps: List[str] field(default_factorylist) # Success criteria met_rto: bool False met_rpo: bool False zero_customer_impact: bool False def calculate_mttr(self) - float: Mean Time To Recovery return (self.end_time - self.start_time).total_seconds() def success_rate(self) - float: Percentage of success criteria met criteria [ self.met_rto, self.met_rpo, self.zero_customer_impact, len(self.alerts_missed) 0 ] return sum(criteria) / len(criteria) * 100 # Example usage metrics GameDayMetrics( scenario_nameDatabase Failover, start_timedatetime(2025, 1, 15, 10, 10, 0), end_timedatetime(2025, 1, 15, 10, 12, 30), time_to_detect_seconds15.0, time_to_respond_seconds45.0, time_to_recover_seconds150.0, error_rate_peak0.05, alerts_fired[DatabaseConnectionError, HighLatency], alerts_missed[FailoverInitiated], responders_involved3, escalations_needed0, met_rtoTrue, met_rpoTrue, zero_customer_impactTrue ) print(fMTTR: {metrics.calculate_mttr()}s) print(fSuccess Rate: {metrics.success_rate()}%)这套数据模型的设计要点GameDayObservation 强调人 时间戳 分类 严重度每条观测都被强制打上category技术 / 流程 / 沟通 / 意外与severity信息 / 关注 / 严重这保证了观察者记录的是结构化事实而非流水账。photo_url字段暗示可附带截图作为时间线证据——这与 Runbook 中Set up screen recording for timelineScreenshot dashboard states的观察者任务互相印证。GameDayMetrics 将技术指标与团队指标并置time_to_detect / time_to_respond / time_to_recover三段时间戳是 SRE 的核心黄金信号alerts_fired与alerts_missed两个列表直接服务于告警准确率这一成功指标responders_involved / escalations_needed / communication_gaps则量化了人的因素。内置两种计算能力calculate_mttr()直接计算平均恢复时间success_rate()用四项成功标准RTO、RPO、零客户影响、零漏报告警求百分比。示例运行会输出MTTR: 150.0s与Success Rate: 75.0%——因为alerts_missed中有FailoverInitiated四项标准只满足三项。这种量化指标 复盘报告的组合与 experiment-design.md 中描述的ChaosExperimentSafety类形成了互补前者在实验进行中做实时监控与自动回滚每 5 秒轮询稳态指标与回滚触发条件后者在演练结束后做沉淀与度量。五、保留惊喜Surprise Scenarios Library惊喜场景是 Game Day 区分于普通故障演练的独特设计——如果所有故障都在预期之内演练就退化成了走流程。原始文档给出了一个必须保密到演练当天的隐藏场景库以下为完整内容。# Keep these secret until game day! surprise_scenarios: - name: Cascading Failure description: Primary failure triggers secondary issue injection: - Database failover (expected) - Cache eviction due to new primary IP (surprise!) learning_goals: - Do we understand our dependencies? - Can we handle multiple simultaneous issues? - name: Monitoring Blind Spot description: Failure that doesnt trigger alerts injection: - Gradual connection pool leak - No immediate alerts fire learning_goals: - How do we discover issues without alerts? - Do we have adequate monitoring coverage? - name: Documentation Failure description: Runbook is outdated or incorrect setup: - Modify runbook to have incorrect commands - Or remove runbook entirely learning_goals: - Can team problem-solve without docs? - How quickly can we update documentation? - name: Key Person Unavailable description: Subject matter expert is unreachable setup: - Ask SME to not respond for 15 minutes learning_goals: - Is knowledge properly distributed? - Can team succeed without specific person? - name: Partial Degradation description: Service works but slowly injection: - Add 5 second latency instead of complete failure learning_goals: - Do we detect performance degradation? - What are our latency SLOs?每个场景的learning_goals都指向一个具体的组织能力缺口而不是为了制造混乱而混乱Cascading Failure检验依赖图谱认知——数据库故障转移后缓存因新主节点 IP 失效团队是否理解修复一个故障会引爆第二个故障的级联效应Monitoring Blind Spot检验监控覆盖度——用逐渐泄漏连接池、不触发任何告警的方式考察团队在没有告警时如何发现异常Documentation Failure检验文档依赖度——故意写错 Runbook 命令或直接删掉 Runbook团队能否脱离文档独立排障Key Person Unavailable检验知识分布——让核心专家 15 分钟不响应团队是否被单点故障卡死Partial Degradation检验部分降级能力——注入 5 秒延迟而非彻底失败考察服务能用但很慢时团队能否识别性能劣化并对照延迟 SLO。从编排角度惊喜场景永远放在常规场景之后Runbook 中位于 11:20前两个已知场景已让团队进入状态并且注入手段要看起来无害但真实——比如 Runbook 中的连接泄漏模拟器通过循环建立 100 个 PostgreSQL 连接且故意不关闭# Intentionally dont close来制造资源耗尽。六、复盘沉淀Post-Game Report 模板演练的价值只有在复盘后才能转化为组织能力。原始文档提供了完整的复盘报告模板用事实指标 结构化分类 行动项取代情绪化总结。以下为完整模板。# Game Day Report: Database Failover **Date**: January 15, 2025 **Participants**: 12 **Duration**: 2 hours **Environment**: Staging ## Executive Summary Conducted database failover game day to test RDS high availability and application resilience. Successfully failed over database in 2.5 minutes (target: 2 min). Discovered 3 critical gaps in monitoring and 2 process improvements needed. ## Metrics | Metric | Target | Actual | Status | |--------|--------|--------|--------| | Time to Detect | 30s | 15s | PASS | | Time to Respond | 5min | 4min 20s | PASS | | Time to Recover | 2min | 2min 30s | FAIL | | Alert Accuracy | 100% | 66% | FAIL | | Zero Customer Impact | Yes | Yes | PASS | ## What Went Well 1. Team responded quickly (4m 20s vs 5m target) 2. Runbooks were accurate and helpful 3. Communication was clear and frequent 4. No customer impact during any scenario 5. Application auto-reconnect worked perfectly ## What Didnt Go Well 1. Missing alert for failover initiation 2. Took 30s longer than target to recover 3. Connection pool exhaustion not detected 4. Dashboard didnt show replica lag clearly 5. Escalation contacts list was outdated ## Surprises 1. Cache invalidation cascaded from DB failover (unexpected) 2. Read replica had 45s replication lag we didnt know about 3. Application retried too aggressively during failover 4. Team found a workaround we hadnt documented ## Action Items | Action | Owner | Due Date | Priority | |--------|-------|----------|----------| | Add alert for RDS failover events | sre-team | Jan 20 | P0 | | Update dashboard with replica lag | platform | Jan 22 | P1 | | Document cache invalidation behavior | dev-team | Jan 25 | P1 | | Add connection pool monitoring | sre-team | Jan 27 | P0 | | Update escalation contact list | manager | Jan 18 | P2 | | Tune application retry backoff | dev-team | Feb 1 | P1 | ## Lessons Learned 1. **Monitoring Gaps**: We had blind spots in replica monitoring 2. **Cascading Effects**: DB changes affect cache in non-obvious ways 3. **Team Knowledge**: Cross-training is working well 4. **Documentation**: Runbooks saved time, keep them updated ## Next Game Day **Proposed Date**: March 15, 2025 **Scenario**: Multi-region failover **Scope**: Production (with safeguards) ## Appendix - Full timeline spreadsheet: [link] - Screen recordings: [link] - Metrics dashboard export: [link] - Raw observation notes: [link]报告模板中最具借鉴价值的两个部分Metrics 表把目标、实际、状态并列Time to Recover 以 2min 30s 对 2min 目标判为 FAILAlert Accuracy 以 66%三枚告警中漏掉一枚判为 FAIL——这不是为了批评团队而是让行动项的优先级P0 的为 RDS 故障转移事件补告警增加连接池监控有了硬数据支撑。Surprises 部分承接惊喜场景的价值报告中的Cache invalidation cascaded from DB failover (unexpected)和Read replica had 45s replication lag we didnt know about正是上文的 Cascading Failure 与 Monitoring Blind Spot 场景挖掘出的真实盲区证明保密惊喜场景的设计确实能产出计划外发现。同时注意报告中的三张表格Metrics / Action Items / Lessons Learned与第四节的数据模型是天然配套的GameDayMetrics中的字段time_to_detect_seconds、alerts_missed等可以自动化地生成 Metrics 表而观察者的GameDayObservation记录则聚合成What Went Well / What Didnt Go Well / Surprises三个章节。这套模板体系事实上构成了一个从量化观测 → 结构化复盘 → 行动项跟踪的闭环。七、与实验设计方法论的衔接稳态、爆炸半径与回滚Game Day 本质上是一组有剧本的混沌实验因此 experiment-design.md 中的方法论可以直接为演练场景的深度设计提供支撑。与本文主题最相关的三点1. 先验证稳态再注入故障。实验模板要求steady_state明确声明指标基线如错误率 0.1%、P99 延迟 500ms、活跃连接数 10并且ChaosExperimentSafety.run_with_safety()在故障注入前会执行 pre-flight 检查——verify_steady_state()逐项校验稳态指标任一越界即中止实验并抛出 System not in steady state - aborting。对应到 Game Day 的 Pre-Game Checklist 就是Verify staging environment healthy。2. 爆炸半径分级与渐进扩大。experiment-design.md 定义了从MINIMAL单实例开发到CRITICAL全量生产的五级爆炸半径并给出关键安全约束CRITICAL 级别必须显式审批生产环境注入比例超过 10% 时必须同时具备 feature flag 与自动回滚单次实验时长超过 600 秒需要审批。其progressive_rollout()函数演示了dev 100% → staging 100% → production 1%的渐进扩大路径。Game Day 的先 staging原则正是这一分级思想在演练组织层面的体现而计划模板中的rollback_plan则是自动回滚 ≤ 30 秒这一安全底线在演练场景中的实例化。3. 单一变量与自动回滚监控。SKILL.md 的安全清单强调每次只改变一个故障条件直到系统行为被充分理解并要求每个实验都必须产出一份书面学习总结和至少一个可追踪的改进项——这与报告模板中 Action Items 表格的 P0/P1/P2 跟踪机制直接对应。而 experiment-design.md 中的monitor_for_rollback()给出了三类回滚触发器时长超限、手动 kill switch、错误率越界的代码化实现值得 Game Day 的自动化工具直接复用。八、演练背后的故障注入工具箱Game Day 的剧本需要故障注入工具来演出。原始文档中的场景使用了 AWS CLIRDS 强制故障转移、安全组断网而 claude-skills 仓库中 chaos-engineer 的其余参考文档为不同场景提供了更丰富的注入手段可直接服务于你的演练剧本设计。基础设施层见 infrastructure-chaos.md# CPU 压测占用 80% 可用核 5 分钟 stress-ng --cpu $(nproc --all) --cpu-load 80 --timeout 5m # 内存压测消耗 70% 可用内存 TOTAL_MEM_MB$(free -m | awk NR2{print $2}) STRESS_MEM_MB$((TOTAL_MEM_MB * 70 / 100)) stress-ng --vm 1 --vm-bytes ${STRESS_MEM_MB}M --timeout 5m # 容器级混沌每 30 秒随机 SIGKILL 匹配 re2:^myapp 的容器 pumba --interval 30s kill --signal SIGKILL re2:^myapp # 容器网络延迟5 分钟 300ms 延迟 50ms 抖动 pumba netem --duration 5m --interface eth0 delay --time 300 --jitter 50 myapp-container网络与依赖层见 chaos-tools.md 与 SKILL.md 中的 Toxiproxy 示例# 为下游依赖创建代理并注入 300ms 延迟 10% 抖动 toxiproxy-cli create -l 0.0.0.0:22222 -u downstream-db:5432 db-proxy toxiproxy-cli toxic add db-proxy -t latency -a latency300 -a jitter30 # 演练结束后移除 toxic 恢复原状 toxiproxy-cli toxic remove db-proxy -n latency_downstream【免费下载链接】claude-skills67 Specialized Skills for Full-Stack Developers. Transform Claude Code into your expert pair programmer.项目地址: https://gitcode.com/GitHub_Trending/claud/claude-skills创作声明：本文部分内容由AI辅助生成（AIGC），仅供参考