Building a Feature Store for Fraud Models

The first fraud model I ever put into production was wrong about a third of the time, and it took us two weeks to figure out why. The model was fine. The features were fine in the notebook. But the numbers the model saw at 3am in production did not match the numbers our data scientist saw when she trained it. A field called avg_txn_amount_7d was computed one way in the training pipeline and a subtly different way in the scoring service. Same name, different math. That mismatch cost us real money in missed chargebacks before anyone noticed.

That bug is the entire reason feature stores exist. Not because they are fashionable, and not because some vendor put "MLOps" on a slide. They exist because the gap between how you compute a signal in training and how you compute it in production is where fraud models quietly die. This is what we built, why, and the parts I would do differently.

The problem is not the model

Everyone new to fraud wants to talk about the model. Gradient boosting versus a neural net, calibration curves, precision at a fixed recall. That is the fun part, and it is also the part that matters least once you are past the demo. In my experience the model is maybe fifteen percent of the work. The other eighty-five percent is getting clean, consistent, timely features into that model at the exact moment a payment is being authorized.

A fraud decision has a hard deadline. When a card is presented, you have roughly a hundred milliseconds of budget to say yes or no before the customer starts to feel it and the acquirer starts to time out. Inside that budget you need to know things like: how many times has this card been seen in the last ten minutes, is this device new to this account, does the billing country match the last twelve transactions. Those are features. Computing them correctly and fast, over and over, without drift, is the whole game.

What a feature store actually is

Strip away the marketing and a feature store is two things stapled together. There is an offline store, usually a data warehouse or a columnar table, where you compute features over historical data to train and backtest. And there is an online store, usually a low-latency key-value system, where the freshest value of each feature sits ready to be read in single-digit milliseconds. The store's real job is to guarantee that both sides compute the same thing.

We use Redis for the online layer and our warehouse for the offline layer. The contract between them is a feature definition: a name, a data type, an entity it belongs to (card, account, device, merchant), and a single transformation that produces it. Write the logic once, run it in both places. That sounds obvious. It is astonishing how many teams still have the training query in a Python notebook and the serving logic hand-ported into a C# service by a different person six months later.

If your training feature and your serving feature are defined in two different files, you do not have a feature store. You have two bugs waiting to disagree.

Training-serving skew is the enemy

The industry term for my 3am bug is training-serving skew, and it is insidious because the model does not crash. It just gets slowly, confidently wrong. The offline pipeline had access to the full day's transactions and computed a seven-day average using a clean window. The online service, under latency pressure, was reading a cached aggregate that only refreshed every fifteen minutes and excluded the current transaction. Small difference. Large consequence.

The fix that finally held was to make the online and offline features come from the same source of truth wherever possible, and to log every feature vector the model actually saw at decision time. That log became gold. When a data scientist retrains, she does not recompute historical features from scratch and hope they match. She trains on the exact vectors production served. Point-in-time correctness, computed once, reused forever. It is slower to set up and it is the single best decision we made.

Point-in-time correctness and the leakage trap

Here is a mistake I have watched three different teams make, including one of mine. You join your labels to your features on the entity key alone, and you accidentally pull in a feature value from after the transaction you are trying to predict. The model looks brilliant in backtest, ninety-eight percent AUC, everyone celebrates. Then it goes live and performs like a coin flip, because in production the future does not exist yet.

Every feature read in the offline store must be as-of the event timestamp, never later. This is why the timestamp is not optional metadata; it is part of the primary key. A correct point-in-time join is more expensive and more annoying to write than a naive one, and it is the difference between a model that works and a very expensive lie. If your backtest looks too good, assume leakage before you assume genius.

-- Point-in-time feature join: only values known AT or BEFORE the event
SELECT
    e.transaction_id,
    e.event_ts,
    f.value        AS avg_txn_amount_7d,
    f.computed_ts
FROM fraud_events e
LEFT JOIN LATERAL (
    SELECT value, computed_ts
    FROM feature_values
    WHERE entity_id  = e.card_id
      AND feature    = 'avg_txn_amount_7d'
      AND computed_ts <= e.event_ts        -- never peek into the future
    ORDER BY computed_ts DESC
    LIMIT 1
) f ON TRUE
WHERE e.event_ts >= '2026-01-01'
  AND f.value IS NOT NULL;

The freshness versus latency fight

Not every feature needs to be fresh to the millisecond, and pretending otherwise will bankrupt your infrastructure. We ended up sorting features into three tiers, and being honest about which tier each one belonged in did more for our latency budget than any clever caching trick.

• Real-time: velocity counts, like transactions per card in the last sixty seconds. These must reflect the current transaction. Computed on the write path, in a streaming job, and read straight from Redis.
• Near-real-time: things like device reputation or rolling seven-day spend. A few minutes stale is fine. We refresh these on a short micro-batch and the cost saving over true streaming is enormous.
• Batch: account age, historical chargeback rate, KYC tier. These change daily at most. Recomputing them per transaction would be absurd.

The trap is treating everything as real-time because it feels safer. It is not safer. It is slower and more fragile, and a streaming pipeline that falls behind gives you stale data with the added bonus of a false sense of freshness. Decide the tier per feature, write it down, and enforce it.

The online serving path

At decision time the scoring service needs a full feature vector fast. Our service reads a batch of keys from Redis in a single pipelined round trip, assembles the vector, and hands it to the model. The whole read budget is around eight milliseconds at the ninety-ninth percentile. Anything that blows that budget gets flagged, because a slow fraud check is functionally a failed one.

The detail that bites people is missing values. In production, a feature will sometimes not be there. A brand new card has no seven-day history. Redis had an eviction. The streaming job hiccuped. Your service cannot throw an exception and reject a legitimate payment because a nice-to-have feature was absent. You need an explicit, versioned default for every feature, and you need to know that the model was trained to handle those defaults the same way.

public FeatureVector Assemble(string cardId, IReadOnlyDictionary<string, double?> raw)
{
    var vector = new FeatureVector();
    foreach (var def in _registry.ActiveFeatures)
    {
        // Missing feature -> explicit trained default, never a silent zero
        var value = raw.TryGetValue(def.Name, out var v) && v.HasValue
            ? v.Value
            : def.DefaultValue;

        vector.Set(def.Name, value);
        if (!raw.ContainsKey(def.Name))
            _metrics.Increment("feature.missing", def.Name);
    }
    return vector;
}

The registry is the product

The piece that surprised me most was how central the feature registry became. It is a boring catalog: every feature, its owner, its type, its tier, its default, the version of its transformation, and when it was last written. But it turned into the thing everyone actually uses. A new analyst can see that device_age_days exists, who owns it, and how it is computed, instead of reinventing a near-identical feature under a new name.

Feature sprawl is real. Within a year we had people creating txn_count_1h and hourly_transaction_count that computed the same thing with slightly different windows. The registry, plus a rule that you cannot ship a feature without registering it, cut the duplication hard. It also gave us lineage for audits, which in regulated payments is not a nice-to-have. When a regulator asks why a customer was declined, "the model said so" is not an answer. "Here are the exact feature values, their sources, and their computation as of that timestamp" is.

Governance, and the decline that matters

Fraud models make decisions that hurt real people when they are wrong. A false positive is not an abstract metric; it is a family whose card gets declined at a checkout while someone stands behind them in line. We treat that with the seriousness it deserves. Every feature that feeds a decision has to be explainable and defensible, which quietly rules out some signals that might squeeze out a bit more accuracy. I would rather ship a slightly less accurate model I can explain than a black box I cannot defend to our compliance team or a regulator.

This is also why the decision log matters as much as the feature log. For every score, we store the vector, the model version, and the top contributing features. When a customer disputes a decline, we can reconstruct exactly what the model saw. That has saved us more than once, both in genuine mistakes we could then fix and in showing that a decline was, in fact, correct.

What I would do differently

I would build the feature logging before the model, not after. We spent months chasing skew that we could have caught on day one if we had simply recorded every production feature vector from the start. I would also resist the urge to build our own store for as long as possible. We hand-rolled ours because the tooling in 2023 felt heavy, and while I do not regret understanding every layer, I have spent engineering time on plumbing that a mature open-source option would have handled.

And I would set a hard cap on feature count earlier. More features feel like progress and mostly are not. Our best model uses about forty features, and the marginal feature past that added latency, added a maintenance burden, and added a new way to silently break. Fewer, well-understood, freshly-served features beat a sprawling zoo every single time.

Conclusion

A fraud model is only as trustworthy as the promise that the number it sees in production is the same number it was trained on. Break that promise and your brilliant model becomes an expensive random number generator that occasionally declines your best customers. Keep it, and a mediocre model earns its keep. The uncomfortable truth is that almost nobody gets famous for building the plumbing that keeps that promise, and almost everybody gets burned when it is missing. So build the boring part first, and be suspicious of any fraud team that would rather show you their model than their feature log.