挑战4-debug_54940393.ppt

执行时间 记录、重播时间 相比于基本执行时间 * 挑战3：调试 * 日志规模 vs.分组规模 分组规模可调试日志规模 * 挑战3：调试 * 数据重播 顺序重播 FT (CLASS=C, NPROCS=64, All-to-ALL) 分组方法 以通信局部性为指导进行分组可有效减小日志 * * CG (CLASS=C, NPROCS=64, Locality) 挑战3：调试 小结 调试HPC程序非常困难 分组再现重播 数据重播和顺序重播的一般形式 记录开销和重播开销的灵活平衡 * 挑战3：调试 * 谢谢！ * Ckpt at MPI API (ckpt can not interrupt sys calls) Adaptive logging/ frontier (two send, one receive) * Umpire Deadlock: Two basic methods exist for deadlock detection in distributed systems: detecting cycles in dependency graphs prior to operation execution, and using timeouts to break the deadlock after it occurs. Umpire uses dependency graphs, although it can gracefully terminate a deadlocked application using timeouts. The Umpire manager tracks blocking MPI communication calls, including collective operations, communicator management routines, and completions of non-blocking requests using its set of MPI history queues for each task. Using this information, Umpire constructs a dependency graph from the ordered operations in each queue. Umpire’s recursive deadlock detector attempts to construct a cycle of dependencies from the blocking operations in these queues. Umpire discards transaction records for these calls when it determines that the operations cannot lead to a deadlock. Realistically, since deadlocks can involve constructs that are not part of MPI, our tool can also time-out after a userspecified period. The timeout period is configurable, so a user can adjust the threshold to accommodate their platform and application. When Umpire times out, it reports the contents of the history queues, as well as * Postmortem trace analysis: collect the trace during normal running, and analysis the trace against interesting issues. Random message payload: try all (or most of) the cases of message order on non-deterministic point. For example, if there are 4 possible messages for one wild-card receive, then on replay, all 4! cases of message orders would be tested. (they named it to test the completeness) * *