ITHICA: Intra-Thread Instruction Checking Approach for Defect-Induced Silent Data Corruptions

Hyperscalers are reporting silent data corruptions (SDCs), presumed to be caused by silicon manufacturing defects, as a threat to datacenter reliability. To support datacenter testing efforts to detect defective CPU servers, this paper presentsITHICA, an approach and tool for automatically generating functional tests for defect-induced errors from arbitrary programs by inserting intra-thread, instruction-level error checks, primarily leveraging instruction duplication and output comparison. Our key insight is that the most pernicious defects (those most likely to escape manufacturing testing) cause inconsistent errors: two executions of the same instruction given the same inputs within the same thread can produce different architectural outputs depending on the execution context in which they run. By exploiting this insight, ITHICA uniquely enables arbitrary programs to serve as tests and localizes affected instructions concurrently with error detection. We use ITHICA to transform industrial hyperscaler tests (our baseline), datacenter programs, and common libraries into functional tests, and evaluate them on over 3,000 CPU servers. ITHICA checks detect 39% more defective servers than baseline industrial checks and yield novel findings on defect behavior that challenge conclusions drawn by prior hyperscaler fleet studies.

• Software systems