Protocol-Sensitive Evaluation of Log Anomaly Detection: Component Costs and Target-Access Sensitivity on HDFS and BGL
Abstract
Published results for log anomaly detection often
depend on how the data are split and which preprocessing
artifacts are available at test time. We examine this depen-
dence by running six pipelines on the Hadoop Distributed
File System (HDFS) and Blue Gene/L (BGL) logs under
several split, parser, temporal, and target-access settings.
The pipelines include three count classifiers, a compact
sequence model, and two classifiers that share a semantic
representation. We also measure parsing, representation
fitting, classifier training, artifact size, and classifier-only
inference as separate components. Random splitting gives
near-ceiling scores on both datasets. When HDFS count-
vector groups are held out intact, or when BGL is split
at a chronological cutoff, the scores fall and the model
ordering changes. In cross-system evaluation, giving the
semantic representation access to an unlabeled target cat-
alog raises XGBoost mean average precision from 0.191
to 0.325 for HDFS-to-BGL and from 0.089 to 0.142 for
BGL-to-HDFS. At a five-percent review budget, the corre-
sponding mean precision and recall are 0.318 and 0.157 in
the first direction and 0.129 and 0.221 in the second. The
study quantifies how protocol and cost boundaries shape
conclusions drawn from these two benchmarks. Code,
split indices, predictions, sensitivity runs, and regenera-
tion checks accompany the submission.
depend on how the data are split and which preprocessing
artifacts are available at test time. We examine this depen-
dence by running six pipelines on the Hadoop Distributed
File System (HDFS) and Blue Gene/L (BGL) logs under
several split, parser, temporal, and target-access settings.
The pipelines include three count classifiers, a compact
sequence model, and two classifiers that share a semantic
representation. We also measure parsing, representation
fitting, classifier training, artifact size, and classifier-only
inference as separate components. Random splitting gives
near-ceiling scores on both datasets. When HDFS count-
vector groups are held out intact, or when BGL is split
at a chronological cutoff, the scores fall and the model
ordering changes. In cross-system evaluation, giving the
semantic representation access to an unlabeled target cat-
alog raises XGBoost mean average precision from 0.191
to 0.325 for HDFS-to-BGL and from 0.089 to 0.142 for
BGL-to-HDFS. At a five-percent review budget, the corre-
sponding mean precision and recall are 0.318 and 0.157 in
the first direction and 0.129 and 0.221 in the second. The
study quantifies how protocol and cost boundaries shape
conclusions drawn from these two benchmarks. Code,
split indices, predictions, sensitivity runs, and regenera-
tion checks accompany the submission.
Research Areas
