Background Jobs in .NET: A Practical Guide

Every payments platform I have worked on eventually grows a second nervous system. Alongside the request-response code that answers an API call in milliseconds, there is a quieter set of work that happens out of band: settlement reconciliation, webhook delivery, statement generation, fraud rescoring, dunning emails. None of it belongs in the hot path of a user request, and almost all of it is where reliability problems hide. Background jobs are how we move that work off the request thread and onto something we can schedule, retry, and observe.

This is a practical guide to running background work in .NET, written from the perspective of someone who has had to explain to a regulator why a batch of settlement files was processed twice. I will cover the built-in primitives, the libraries worth reaching for, and the operational details that separate a demo from a system you can leave running over a long weekend.

What Actually Counts as a Background Job

The term "background job" gets stretched to cover three quite different things, and conflating them is the root of a lot of pain. The first is in-process deferred work: something triggered by a request that you do not want the caller to wait for, like firing off a confirmation email. The second is scheduled work: recurring tasks that run on a clock, such as a nightly reconciliation run. The third is durable queued work: tasks that must survive a process restart and be processed exactly once, like applying a payment to a ledger.

These categories have wildly different durability requirements. The confirmation email can be lost on a deploy and nobody will be harmed; the ledger posting cannot. When I review a design, the first question I ask is which of these three a piece of work belongs to, because the answer dictates everything that follows. Treating a durable financial operation as fire-and-forget in-process work is the kind of mistake that does not show up in testing and shows up very loudly in production.

A useful rule of thumb: if losing the job would require a manual correction or a customer apology, it needs durable storage outside the process. If losing it is merely annoying, you have far more freedom in how you implement it.

Hosted Services Are the Foundation

The base abstraction in modern .NET is IHostedService, and more practically its convenience base class BackgroundService. The generic host manages the lifecycle: it calls your start logic when the application boots and signals a cancellation token when it is shutting down. Almost everything else in this article is, under the hood, a hosted service of some kind.

The pattern most people get wrong is shutdown. The host gives you a cancellation token; you must honour it, both to stop pulling new work and to give in-flight work a chance to finish within the shutdown timeout. A long-running loop that ignores the token will be killed mid-operation, which for financial work means a half-written state you then have to reason about.

public class SettlementWorker : BackgroundService
{
    private readonly IServiceScopeFactory _scopes;
    private readonly ILogger<SettlementWorker> _logger;

    public SettlementWorker(IServiceScopeFactory scopes, ILogger<SettlementWorker> logger)
    {
        _scopes = scopes;
        _logger = logger;
    }

    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        while (!stoppingToken.IsCancellationRequested)
        {
            using var scope = _scopes.CreateScope();
            var processor = scope.ServiceProvider.GetRequiredService<ISettlementProcessor>();

            try
            {
                await processor.ProcessPendingAsync(stoppingToken);
            }
            catch (OperationCanceledException) when (stoppingToken.IsCancellationRequested)
            {
                break; // clean shutdown, not an error
            }
            catch (Exception ex)
            {
                _logger.LogError(ex, "Settlement batch failed; will retry next cycle");
            }

            await Task.Delay(TimeSpan.FromSeconds(30), stoppingToken);
        }
    }
}

Note the scope creation inside the loop. A BackgroundService is a singleton, so it cannot directly hold a scoped dependency like a database context. Creating a scope per iteration mirrors how a web request would resolve services and avoids the subtle bug of a single long-lived DbContext accumulating tracked entities for the lifetime of the process.

When to Reach for a Library

The raw hosted service is fine for polling loops and simple timers. The moment you need durable, retryable, observable jobs, writing your own queue is a poor use of time. The .NET ecosystem has mature options, and choosing between them is mostly about where you want the durability to live.

• Hangfire stores jobs in your existing database (SQL Server, PostgreSQL, and others) and ships with a built-in dashboard. It is the path of least resistance when you already have a relational database and want recurring jobs, delayed jobs, and retries without standing up new infrastructure.
• Quartz.NET is the heavyweight scheduler, with cron-style triggers, clustering, and misfire handling. I reach for it when scheduling semantics are genuinely complex, for example calendars that skip bank holidays.
• MassTransit or Wolverine over a real broker (RabbitMQ, Azure Service Bus, Amazon SQS) is the right answer when jobs are really messages flowing between services, and you want the broker to own delivery guarantees rather than your database.

I have shipped all three in anger. My default for a single-service application with a database already in place is Hangfire, because the operational surface is small and the dashboard alone pays for itself during incidents. When the work is inherently distributed across services, I move the durability to a broker and treat jobs as messages, because trying to coordinate cross-service work through a shared jobs table tends to leak the database as an integration point.

Idempotency Is Not Optional

Every background job system you will use guarantees at-least-once execution, not exactly-once. A worker can crash after doing the work but before recording that it finished, and the job will run again. This is not a flaw in any particular library; it is a consequence of distributed systems and the fact that "do the work" and "mark it done" are two separate operations that cannot be made atomic across a network.

The practical consequence is that your job handlers must be idempotent. Running the same job twice must produce the same result as running it once. For a ledger posting, that means deriving a deterministic idempotency key from the business event and enforcing uniqueness at the database level, so the second attempt fails the insert rather than double-posting.

If you cannot articulate what happens when a job runs twice, you have not finished designing it. Assume every job will eventually run twice, because in production it will.

The cheapest enforcement mechanism is usually a unique constraint. Insert a row keyed on the natural identity of the operation before doing anything else; if the insert violates the constraint, the work is already done or in flight, and you can stop. This pushes correctness into the database, which is exactly where you want it for money movement, rather than relying on application logic that a future refactor might quietly break.

Retries, Backoff, and Poison Messages

Retries are the feature people enable first and reason about least. A transient failure, a brief network blip to a payment processor, deserves a retry. A permanent failure, a malformed record that will never parse, does not; retrying it forever just burns capacity and fills your logs. The discipline is to distinguish the two and treat them differently.

For transient failures I use exponential backoff with jitter. Backoff stops you from hammering a struggling dependency, and jitter stops every worker in the fleet from retrying in lockstep and creating a thundering herd. Most libraries give you backoff out of the box; if you are rolling your own, Polly is the standard policy library and integrates cleanly with the hosted service model.

The job that exhausts its retries needs somewhere to go. A dead-letter queue or a failed-jobs table is essential, because the alternative is a job that silently vanishes. In a regulated environment, silent loss is the worst outcome: you would rather an operator be paged to inspect a stuck settlement than discover three days later that a file was never processed. I treat the dead-letter queue as a first-class alerting signal, not a dumping ground I check when I remember.

Concurrency and Ordering Control

Background workers scale by running more of them, but unbounded concurrency is its own hazard. Two workers picking up the same job, or several workers updating the same account balance at once, will corrupt state unless you control it. The two levers are how you claim work and how you serialise work that touches the same resource.

Claiming work safely means an atomic dequeue. In a database-backed system, this typically uses row locking with skip-locked semantics so that each worker grabs a distinct row without blocking the others. PostgreSQL makes this clean:

UPDATE job_queue
SET status = 'processing',
    locked_by = @workerId,
    locked_at = now()
WHERE id = (
    SELECT id FROM job_queue
    WHERE status = 'pending'
    ORDER BY created_at
    FOR UPDATE SKIP LOCKED
    LIMIT 1
)
RETURNING id, payload;

Ordering is the subtler problem. Most background work does not need global ordering, but some does: you cannot apply a refund before the original charge has posted. The cleanest approach I have found is partitioned ordering, where work is keyed by some entity, an account or a merchant, and the system guarantees order within a partition while allowing parallelism across partitions. Brokers like Azure Service Bus expose this directly through session-based ordering, which is far more robust than trying to bolt sequencing onto an unordered queue after the fact.

Observability and the Dashboard

Background jobs fail in the dark. A web endpoint that breaks shows up immediately in error rates and customer complaints; a job that quietly stops running can go unnoticed until a downstream report comes out wrong days later. Observability is therefore not a nice-to-have for background work, it is the difference between a controlled system and a black box.

I instrument three things on every job system. The first is queue depth and age of the oldest pending job, which tells me whether I am keeping up. The second is success, failure, and retry rates per job type, emitted as metrics so I can alert on them. The third is structured logs that carry the job id and the business identity through every attempt, so when something does fail I can trace exactly what happened without reconstructing it from fragments.

The Hangfire dashboard, or an equivalent view into your broker, earns its keep during incidents. Being able to see the failed-jobs list, read the exception, and requeue a corrected job from a web UI turns a midnight investigation into a five-minute task. But a dashboard is a debugging tool, not a monitoring strategy. The alerts that wake someone up should come from metrics, not from a human happening to look at a screen.

Deployment and Graceful Shutdown

Most background job incidents I have investigated trace back to a deploy. When a process is told to stop, in-flight jobs are at their most vulnerable. If the host kills the process before a job finishes and the job is not idempotent, you get the double-processing scenario we already discussed. The defence is graceful shutdown that actually works, end to end.

This means the host shutdown timeout must be long enough for a typical job to drain, the worker must stop claiming new work as soon as the cancellation token fires, and the orchestrator, whether Kubernetes or a Windows service manager, must give the process that grace period before sending a hard kill. Each of these is configured in a different place, and they only protect you if all three agree. I have seen a perfectly written shutdown handler defeated by a thirty-second Kubernetes termination grace period that was shorter than the job it was trying to protect.

A related question is whether to co-locate workers with your web process or run them separately. For small systems, co-location is simpler. As load grows, I split them, because background work and request handling have different scaling profiles and different failure modes, and a runaway batch job should never be able to starve the threads serving live customers.

Conclusion

Background jobs are deceptively easy to start and genuinely hard to run well. The .NET platform gives you a solid foundation in the generic host and a mature set of libraries on top of it, so the technology is rarely the limiting factor. What separates a reliable system is the discipline around it: classifying work by its durability needs, making every handler idempotent, distinguishing transient from permanent failures, controlling concurrency, and instrumenting everything so that failures surface loudly rather than silently. Get those right and the choice between Hangfire, Quartz, and a broker becomes a matter of taste rather than survival. In a regulated environment, that discipline is not overhead; it is the whole job.