Beyond Capability Benchmarks: Learning Operational Fingerprints of LLM Cloud Services from Production Incident Metadata

Managed LLM services are now part of real production systems, but model selection and service planning still rely heavily on capability benchmarks that reveal little about operational behavior after deployment. We present Operational Embedding (OpEmbed), a framework for learning compact operational fingerprints of LLM cloud services from structured support-ticket metadata. OpEmbed aggregates model--time windows into multi-channel operational signatures and learns a low-dimensional representation with temporal contrastive learning, cross-view reconstruction, and ordinal regularization. Across more than 33,000 production support cases spanning multiple model families and 26 months, OpEmbed recovers stable family- and version-level structure, improves leave-one-model-out forecasting over raw-signature and simple family/global baselines, remains useful under limited early-window data, and supports cross-model fault transfer with lower divergence and higher Top-K hit rates. These results suggest that production metadata can support practical workflows such as model onboarding, support readiness assessment, fault triage, and operational monitoring, providing an industry-oriented complement to capability-centric LLM evaluation.