The incident
We were building a service-to-service integration between our ASP.NET Core API and a third-party financial data platform. The integration used JWT Bearer authentication — we'd generate a short-lived token signed with our RSA private key, and the third party's API would validate it against our published public key (via JWKS endpoint). This is a standard OAuth 2.0 client credentials flow, and it worked perfectly in staging.
In production, we started seeing intermittent 401 Unauthorized errors. Not consistent — maybe one in every fifty calls. The tokens looked valid when decoded. The claims were correct. The signature checked out. We were generating new tokens for every request, so it wasn't an expiry issue. For three days, we couldn't reproduce the failure in any controlled environment, and the third party's API returned nothing more useful than "401 Unauthorized" with no body.
What we tried first — and why it failed
Our first assumption was a signing key issue. We regenerated the RSA key pair, republished the JWKS endpoint, and updated the private key in Azure Key Vault. The 401s continued at the same rate. We then suspected a race condition in our token generation code — the secret was being read from configuration while another thread was refreshing it. We added locking around the key access. Still the same error rate.
We also tried adding extensive logging around every token we generated: the claims, the serialized token string, the timestamp. When we manually decoded the failing tokens from our logs using the JWT Decoder, every single one looked structurally correct. The nbf (not before) was set to the current UTC time, exp was 10 minutes later, and the claims were accurate.
Finding the real cause
The breakthrough came when the third party's support team finally gave us the raw error message from their internal logs: "IDX10222: Lifetime validation failed. The token is not yet valid. ValidFrom: 2026-06-12T09:17:43Z, Current time: 2026-06-12T09:17:10Z."
Our token said it was valid from 9:17:43 UTC. Their server's clock said it was currently 9:17:10 UTC — 33 seconds earlier. From their perspective, the token hadn't become valid yet. This happened intermittently because our Azure VM's system clock was drifting ahead. We verified this by comparing DateTimeOffset.UtcNow from our service against an external NTP time source: our clock was running between 30 and 90 seconds fast, with the drift varying as the VM's hypervisor time synchronisation kicked in and out.
The third party's validator had ClockSkew = TimeSpan.Zero. That's the key detail. ASP.NET Core's default ClockSkew in JwtBearerOptions is 5 minutes — a deliberate safety margin that exists specifically to absorb this kind of clock drift between distributed systems. The third party had removed this protection. Their validator was strict to the second, which is technically correct per RFC 7519, but practically fragile in a cloud environment where VM clocks drift.
What our original token generation looked like
// Original code — generated tokens that failed intermittently
public string GenerateServiceToken()
{
using var rsa = RSA.Create();
rsa.ImportFromPem(_config["Jwt:PrivateKeyPem"]);
var credentials = new SigningCredentials(
new RsaSecurityKey(rsa), SecurityAlgorithms.RsaSha256);
var now = DateTime.UtcNow; // <-- problem: single point-in-time on a drifting clock
var descriptor = new SecurityTokenDescriptor
{
Issuer = "https://api.ourservice.example.com",
Subject = new ClaimsIdentity(new[] {
new Claim("sub", "our-service-client-id")
}),
IssuedAt = now,
NotBefore = now, // <-- nbf = exact current time on our drifting clock
Expires = now.AddMinutes(10),
SigningCredentials = credentials
};
return new JwtSecurityTokenHandler().WriteToken(
new JwtSecurityTokenHandler().CreateToken(descriptor));
}The problem is NotBefore = now where now is our server's clock. If our clock is 73 seconds ahead of the recipient's clock, and their ClockSkew is zero, our token arrives in their past (from their perspective) and is rejected as "not yet valid" until their clock catches up.
The fix
// Fixed code — backdates nbf to absorb clock skew on the receiving end
public string GenerateServiceToken()
{
using var rsa = RSA.Create();
rsa.ImportFromPem(_config["Jwt:PrivateKeyPem"]);
var credentials = new SigningCredentials(
new RsaSecurityKey(rsa), SecurityAlgorithms.RsaSha256);
var now = DateTimeOffset.UtcNow; // DateTimeOffset is more explicit about timezone
var descriptor = new SecurityTokenDescriptor
{
Issuer = "https://api.ourservice.example.com",
Subject = new ClaimsIdentity(new[] {
new Claim("sub", "our-service-client-id")
}),
IssuedAt = now.UtcDateTime,
NotBefore = now.AddSeconds(-120).UtcDateTime, // 2-minute tolerance window
Expires = now.AddMinutes(10).UtcDateTime,
SigningCredentials = credentials
};
return new JwtSecurityTokenHandler().WriteToken(
new JwtSecurityTokenHandler().CreateToken(descriptor));
}By setting NotBefore to 120 seconds in the past, we tell recipients: "This token became valid 2 minutes ago." Even if our clock is 90 seconds ahead of theirs, from their perspective the nbf is still in their past, and the token is accepted. We don't change Expires — the token's effective validity window is still 10 minutes from when we generate it.
We also added the following on our ASP.NET Core receiving side (for tokens we accept from other services), restoring the default ClockSkew rather than fighting it:
// Don't zero out ClockSkew — the 5-minute default exists for a reason
builder.Services.AddAuthentication(JwtBearerDefaults.AuthenticationScheme)
.AddJwtBearer(options =>
{
options.TokenValidationParameters = new TokenValidationParameters
{
ValidateIssuer = true,
ValidIssuer = "https://trusted-service.example.com",
ValidateAudience = true,
ValidAudience = "our-api",
ValidateLifetime = true,
ClockSkew = TimeSpan.FromMinutes(2), // explicit 2-minute tolerance
IssuerSigningKeyResolver = (_, _, _, _) => GetCurrentPublicKeys()
};
options.Events = new JwtBearerEvents
{
OnAuthenticationFailed = ctx =>
{
// Log the full exception — the Message property names the specific
// validation failure (lifetime, issuer, audience, signature)
_logger.LogWarning("JWT validation failed: {ExceptionType}: {Message}",
ctx.Exception.GetType().Name, ctx.Exception.Message);
return Task.CompletedTask;
}
};
});What I've changed permanently
After this incident I added OnAuthenticationFailed logging to every ASP.NET Core JWT configuration in our codebase. The default behaviour swallows the exception and returns a generic 401 — which is exactly why this took three days to diagnose. Logging the specific exception type and message would have shown "IDX10222: Lifetime validation failed" immediately, and we'd have had a lead within hours.
I also stopped setting ClockSkew = TimeSpan.Zero by default. The temptation is to be strict, and strictness feels secure. But in practice, clock skew between distributed systems is real, ongoing, and impossible to fully control in cloud environments. The 5-minute default isn't a security weakness — it's an engineering acknowledgment that clocks drift. If your security model genuinely requires zero tolerance (e.g., short-lived one-time-use tokens), you need a different mechanism than clock-based validation.
The generalizable lesson
This bug belongs to a class of distributed systems failures I'd call "locally impossible, distributed inevitable." You cannot reproduce clock skew in a single-machine test environment. The failure only manifests when two separate systems with independent clocks interact, and the drift accumulates to a value larger than your tolerance window. Every piece of code that touches timestamps in a distributed context needs to account for this explicitly.
Before deploying any JWT-based service-to-service integration, verify the clock drift on both ends using an external NTP source. Azure VMs use the Windows Time service by default, but this can drift by minutes in some VM configurations. For critical integrations, consider a short (5-10 second) sanity check that compares your server's DateTimeOffset.UtcNow against a known reference time before generating security-critical tokens.
Use the JWT Decoder to inspect any failing token and compare the nbf and exp values against actual UTC time. If the token looks valid by eye but keeps being rejected, clock skew should be your first hypothesis, not your last.