A Realistic Benchmark Dataset for Multi-Model Resume Screening and Candidate Ranking

Large Language Models (LLMs) are increasingly deployed to automate enterprise recruitment pipelines, yet evaluating their decision consistency and underlying logic remains heavily throttled by a lack of credible, real-world benchmark datasets. Free-form floating-point scoring methods frequently used in recruitment AI suffer from severe numerical calibration drift and mask the models' intermediate reasoning. To address these challenges, we introduce Djinni-CB, a novel, two-tiered evaluation benchmark containing 2,484 real candidate-job pairs spanning 21 diverse job categories, bridged entirely from naturally occurring, anonymized candidate résumés and standardized employer postings.
Rather than relying on generic scores, we implement a rigorous 29-question structured questionnaire across 9 diagnostic groups to audit pure deterministic reasoning boundaries. We establish comprehensive baseline results at temperature T=0.0 comparing frontier model architectures: Gemini 4.1m, DeepSeek Pro, GPT-5.4, and Gemini 2.5 Pro. Over 235,000 discrete decision points reveal a distinct dichotomy in the LLM ecosystem: a "Literal Gatekeeper" cluster (DeepSeek Pro and GPT-5.4) that prioritizes strict keyword matching, and a "Holistic" cluster (the Gemini family) that leverages contextual domain inference. Our dataset provides a highly structured resource complete with verbatim natural-language reasoning logs, offering an essential framework for future research into interpretability, factuality, and consistency in automated hiring.