Predicting the number of fatal soft errors in los alamos national labratory’s ASC Q comput.pdf

IEEE TRANSACTIONS ON DEVICE AND MATERIALS RELIABILITY, VOL. 5, NO. 3, SEPTEMBER 2005 329 Predicting the Number of Fatal Soft Errors in Los Alamos National Laboratory’s ASC Q Supercomputer Sarah E. Michalak, Kevin W. Harris, Member, IEEE, Nicolas W. Hengartner, Bruce E. Takala, and Stephen A. Wender Invited Paper Abstract—Early in the deployment of the Advanced Simulation separate CPU boards. Further information about Q’s architec- and Computing (ASC) Q supercomputer, a higher-than-expected ture is available in the Appendix. number of single-node failures was observed. The elevated rate of Q is used for scientific computing, jobs that typically require single-node failures was hypothesized to be caused primarily by fatal soft errors, i.e., board-level cache (B-cache) tag (BTAG) par- many processors for many hours, and is shared by Los Alamos, ity errors caused by cosmic-ray-induced neutrons that led to node Livermore, and Sandia National Laboratories. A large calcu- crashes. A series of experiments was undertaken at the Los Alamos lation is performed on Q by dividing it into pieces, each of Neutron Science Center (LANSCE) to ascertain whether fatal which is calculated on a different processor. Performing a soft errors were indeed the primary cause of the elevated rate of calculation in this manner requires interprocessor communica- single-node failures. Observed failure data from Q are consis- tent with the results from some of these experiments. Mitigation tion using message passing interface [