One of the most interesting challenges I have worked on involved helping transform fraud and security detection from a reactive operation into a proactive intelligence capability.
The work focused on defending retail banking infrastructure against credential stuffing, bot-driven account takeover attempts, and increasingly sophisticated low-and-slow attack campaigns. The environment generated large volumes of security signals from edge systems, banking platforms, and infrastructure monitoring tools. At the same time, the operational teams consuming the output had little tolerance for false positives.
The stakeholders were diverse. Security Operations analysts needed alerts they could trust. Fraud analytics teams needed broader threat visibility. Engineering leadership required a solution that could operate entirely within regulated infrastructure boundaries. Executive stakeholders wanted assurance that the organization could detect emerging threats without creating friction for legitimate customers.
The challenge was not simply building a machine learning model. It was designing a system capable of operating under real-world constraints: highly imbalanced data, evolving adversaries, strict security controls, and enterprise-scale production requirements.
The Problem Looked Simple
The original problem appeared straightforward: improve detection of malicious activity that traditional security controls were missing.
In practice, it was significantly more complicated.
Retail banking environments are constantly targeted by attackers who use residential proxy networks, rotating infrastructure, dynamic user agents, and automated tooling designed to mimic legitimate user behavior. Traditional rate-limiting and IP-based blocking rules were increasingly ineffective because attackers no longer behaved like obvious bots.
A second challenge was the underlying class imbalance. Fraudulent events represented less than 0.5% of all observed traffic. In that environment, even highly accurate models can create operational problems if they generate excessive false positives.
I initially explored a supervised machine learning approach using XGBoost. Offline results were excellent. Precision exceeded 99%, creating an early sense that the problem was largely solved. However, once testing expanded beyond historical examples, an uncomfortable reality emerged.
The model performed exceptionally well at detecting known attack signatures, yet its performance deteriorated when encountering novel bot behaviors. Recall dropped into the 88% to 92% range because the model had learned patterns from the past rather than identifying emerging threats.
The natural alternative was anomaly detection. Isolation Forest successfully surfaced many previously unseen attack patterns and achieved substantially higher capture rates. Unfortunately, it created a different problem. Legitimate high-volume users, including corporate banking customers and power users, frequently resembled anomalies. Precision fell into the 40% to 50% range, producing far more alerts than analysts could reasonably investigate.
At that point it became clear that neither approach, on its own, could satisfy both business and operational requirements.
Challenging Early Assumptions
One lesson from this project was that strong notebook metrics do not guarantee production success.
At the beginning, I believed a sufficiently trained supervised model would provide the majority of the value. Historical labels were available, feature importance looked sensible, and evaluation metrics were strong.
The data ultimately proved that assumption wrong.
As testing expanded, I repeatedly encountered examples of attack behaviors that had never appeared in training data. The model was not failing because it was poorly built. It was failing because attackers evolve faster than historical labels.
That realization fundamentally changed the direction of the project.
Instead of trying to maximize performance from a single model, I started reframing the problem around complementary strengths. The goal became combining a model that excelled at recognizing known fraud patterns with another model designed to surface unexpected behavior.
Feature Engineering and Behavioral Analysis
A large portion of the effort focused on understanding behavior rather than individual events.
We expanded the feature set into a collection of behavioral indicators covering login activity, request volume, infrastructure signals, ASN fraud indicators, device churn patterns, and low-and-slow activity characteristics. Rather than asking whether a single event looked malicious, the analysis focused on how entities behaved across time and infrastructure dimensions.
What took much longer than expected was determining which signals represented genuine attack behavior versus legitimate operational variance.
This became especially apparent when analyzing customers with unusually high session volume. Traditional anomaly detection approaches consistently treated these users as suspicious because their activity sat on the edge of the distribution.
However, detailed investigation revealed many of these records belonged to legitimate enterprise users whose behavior, while uncommon, was entirely expected.
The challenge was finding a way to reduce those false positives without suppressing actual attacks.
Discovering the Calibration Problem
One of the more interesting technical problems had no obvious textbook solution.
Initially, I experimented with combining outputs from the supervised and unsupervised models by directly averaging their scores. On paper, the approach seemed reasonable.
In practice, it failed.
The root issue was that the models produced fundamentally different outputs. XGBoost generated calibrated probabilities, while Isolation Forest produced anomaly scores with different statistical properties. Averaging them introduced instability and inconsistent rankings because the score ranges were not truly comparable.
The breakthrough came from focusing on relative ranking instead of raw scores.
By converting both outputs into percentile rankings, I could normalize their influence without requiring complex probability calibration. This approach preserved the ordering strength of each model while eliminating scale mismatch issues.
Production Migration Challenges
While model development was important, the migration from research to production ultimately consumed more time.
The original prototypes lived in local notebook environments. Production execution required operating entirely within the bank's governed data platform under strict data-governance requirements. Customer data could not leave the protected environment, which meant both scoring and orchestration needed to run inside approved infrastructure.
Several implementation details became surprisingly significant.
One recurring issue involved schema alignment. Training datasets used data-science-friendly feature naming conventions, while production tables used enterprise-standard uppercase schemas. Small inconsistencies in naming created inference failures, signature mismatches, and deployment issues.
I eventually standardized feature mapping through explicit schema translation layers rather than relying on implicit assumptions. It was not the most glamorous engineering work, but it dramatically improved reliability.
Another challenge involved model governance and registry integration. Registering composite model artifacts exposed several undocumented behaviors and version-specific incompatibilities. What seemed like a straightforward deployment task evolved into a series of type mismatches, serialization issues, and artifact-handling failures.
That experience reinforced a principle I frequently explain to stakeholders: production machine learning rarely fails because of algorithms. It normally fails because of interfaces, dependencies, governance requirements, and operational assumptions.
The Architecture We Shipped
The final architecture adopted a layered ensemble strategy designed to balance precision, recall, and operational usability.
The core detection engine combined an XGBoost classifier optimized for known threat patterns and an Isolation Forest model designed to identify novel behavioral anomalies. The models served different purposes. XGBoost provided highly precise identification of attack signatures already represented in historical data. Isolation Forest provided exploration capability by surfacing suspicious behaviors that resembled nothing seen previously.
Rather than treating them as equal contributors, I implemented a weighted rank-sum ensemble with a 70/30 weighting in favor of XGBoost. The weighting reflected the operational reality that precision remained critically important while still gaining the recall benefits of anomaly detection.
The ensemble converted model outputs into percentile rankings before blending. This design decision addressed several challenges at once: elimination of score calibration mismatches, greater resilience to distribution shifts, consistent prioritization across model families, and improved interpretability for downstream consumers. Instead of debating whether one score of 0.82 should outweigh another score of 0.67, the system focused on where observations ranked relative to their peers.
Through repeated investigation of analyst triage results, I observed that many high-volume users generated anomalous activity patterns without showing associated infrastructure risk indicators. Rather than increasing global thresholds and sacrificing recall, I introduced a conditional attenuation mechanism. When specific behavioral signals suggested trusted high-volume usage without corresponding infrastructure anomalies, the risk score was substantially reduced.
This was a deliberate trade-off. The objective was not to suppress alerts permanently. Instead, it was to reduce noise while preserving visibility if supporting risk signals began to emerge.
Production reliability required several additional design decisions. For model serialization, I moved XGBoost artifacts to UBJSON format instead of traditional pickle files. The change improved portability, reduced environment dependency issues, and eliminated several governance concerns associated with executable serialization formats.
For anomaly detection, I separated StandardScaler fitting from production inference. Training statistics were frozen and reused during scoring. This prevented attackers from inadvertently influencing normalization parameters during periods of unusual traffic volume. Managed orchestration and in-platform execution completed the operational architecture, allowing feature extraction, scoring, ranking, and queue generation to run as production workflows entirely within governed infrastructure.
Outcome
The final ensemble significantly outperformed either standalone approach.
Configuration | Precision | Recall |
|---|---|---|
XGBoost alone | 99.5% | 88% to 92% on novel attacks |
Isolation Forest alone | 40% to 50% | 98.5% |
Rank-blended ensemble | 96.2% | 97.8% |
The operational impact was equally important. At the highest-risk percentile queue (top 0.1%), analysts reviewed only 106 records while identifying 38 confirmed fraud instances, yielding a precision hit rate of 35.85%. This dramatically concentrated investigative effort where it mattered most.
These figures come from a single evaluation period, not a sustained production average. The 96.2% ensemble precision and the 35.85% queue hit rate also measure different things: labeled evaluation precision versus analyst-confirmed outcomes in a prioritized review queue, where a record can remain genuinely high-risk and still close without a fraud disposition.
Beyond model performance, the project migrated from isolated notebook experimentation into a governed enterprise architecture with model registry integration, in-platform execution, and orchestrated production workflows.
What I Took Away
Several lessons from this project continue to influence how I approach machine learning systems.
First, supervised learning is rarely sufficient in adversarial environments. Historical labels are valuable, but attackers continuously innovate. Effective defense requires balancing known-pattern detection with mechanisms for surfacing genuinely new behavior.
Second, operational usability matters as much as model accuracy. A model that catches everything but overwhelms analysts with noise is not a successful solution. Detection systems must optimize for decision support, not simply metrics.
Third, data preprocessing deserves the same rigor as model selection. The discovery that dynamic scaling could distort anomaly distributions under attack conditions reinforced how easily production behavior can diverge from laboratory assumptions.
Finally, production machine learning is fundamentally a systems problem. Architecture, governance, deployment, schema management, and operational resilience often determine success more than model choice itself. Building a technically strong model is only the beginning. Building one that remains reliable, explainable, maintainable, and trusted in production is where the real work begins.
The success of this build came from acknowledging that no single model could solve the problem. By combining supervised precision, unsupervised discovery, rank-based ensemble logic, and targeted false-positive controls, I was able to create a detection system that improved threat capture while remaining operationally practical. The approach worked because it balanced statistical performance with the realities of enterprise security operations, producing a solution that was not only accurate, but deployable, governable, and trusted.
If you are working through a similar trade-off between notebook precision and analyst-ready detection, our team at Human Managed would welcome the conversation.
