《QCon上海2015-大型分布式系统设计的一些黄金原则和实例-俞圆圆》.pdf

大型分布式系统设计的一些黄金原则和实例 俞圆圆 a.k.a Y3 YuanYuan.Yu@UC Agenda • 4个在生产环境中的1, 2级事 • 故障排查的过程和根因分析 • 反思和教训 4 案例1：.NET框架里的定时炸弹 5 背景情况 • 对接Azure官网的前端服务(Frontend Service Fleet) • 非高频请求（~500万请求/ 日) 官网入口 Frontend Frontend Frontend Service Service Service API 层 6 事故现象 • 大约3天前发布了新的v6.0.1版frontend service • 没有任何告警，所有监控runner状态均为绿色 • 但前端团队反应有间歇性请求超时 • 典型的 “brown out” • 请求数量没有明显增加 • 最奇怪的地方：新的v6.0.1版已经发布数天了？! 7 故障排查：Frontend Service的CPU Counter 8 根因：.NET类库中的bug • Process dump显示大量的对CPU的占用均来自于一个API: ListClientRootCertificates • .NET类库里X509Certificates2类的构造函数里有一个临时 文件句柄泄露的bug: - /kb/931908 - 最高可用临时文件数：65000 9 反思和教训：什么是好的监控？ • Sign-off and Forget it does not work. Continuous monitoring/alerting is a must. • Features that improve monitoring, debugging and operational efficiency are as important as performance. - Runner only raises alert if 5 consecutive calls fail (5-minute span), not good enough - Design your monitoring like how you customer will use your service • 好的监控 ：actionable, unambiguous, dumb … In retrospect, this is more of a monitoring miss than insuffici ent test coverage. Alerts that are non-actionable, have multip le interpretations, and are too sensitive/insensitive, are worse than no alerts. It is inevitable to have such edge cases that