Implement gRFC A97: xDS JWT Call Credentials - #12951
Conversation
shivaspeaks
left a comment
There was a problem hiding this comment.
Sending what I have with the high level review. I need to review it in-depth.
| } | ||
|
|
||
| if (readState == ReadState.BACKOFF) { | ||
| failStatus = lastReadFailureStatus != null ? lastReadFailureStatus : Status.UNAVAILABLE; |
There was a problem hiding this comment.
The gRFC says,
If a data plane RPC is started when there is no cached token available and while in backoff delay, it will be failed with the status from the last attempt to read the file.
So this should come after we check hasValidCache check, so that the BACKOFF state only fails the RPC if there is NO valid cached token.
| } | ||
| byte[] payloadBytes; | ||
| try { | ||
| payloadBytes = com.google.common.io.BaseEncoding.base64Url() |
There was a problem hiding this comment.
Use import statements here and other places for readability. (Probably we can add this as a rule to take a pass for these nits)
| @VisibleForTesting | ||
| static boolean enableXdsFallback = GrpcUtil.getFlag(GRPC_EXPERIMENTAL_XDS_FALLBACK, true); | ||
|
|
||
| @VisibleForTesting |
There was a problem hiding this comment.
Right now this serves no purpose because you're using reflection to set it.
| throw new XdsInitializationException( | ||
| "Invalid bootstrap: server " + serverUri + " with 'jwt_token_file' config missing"); | ||
| } | ||
| String tokenFile = JsonUtil.getString(config, "token_file"); |
There was a problem hiding this comment.
Incorrectly attempts to read "token_file" from config. Should be "jwt_token_file". I'll also recommend to change the variable name to jwtTokenFile.
| } catch (Exception e) { | ||
| throw new RuntimeException(e); | ||
| } | ||
| } |
There was a problem hiding this comment.
It is okay to use reflection but there is a simpler way. Reflection is inherently slower than direct field access because the JVM has to do extra work at runtime.
I can assume why you made this choice because enableXdsBootstrapCallCreds in BootstrapperImpl.java is not public, hence you chose to go with reflection. We can safely make it public there because it is anyway internal.
| + " {\n" | ||
| + " \"type\": \"jwt_token_file\",\n" | ||
| + " \"config\": {\n" | ||
| + " \"token_file\": \"/var/run/secrets/token\"\n" |
There was a problem hiding this comment.
jwt_token_file* here and below in this file.
| ImmutableMap.of( | ||
| "type", "jwt_token_file", | ||
| "config", ImmutableMap.of( | ||
| "token_file", jwtTokenFile.toAbsolutePath().toString()) |
|
/gemini review |
There was a problem hiding this comment.
Code Review
This pull request introduces JwtTokenFileCallCredentials to load, parse, cache, and refresh JWT tokens from a file, and integrates it into the xDS bootstrap configuration to support call_creds. The review feedback identifies two bugs in JwtTokenFileCallCredentials: a critical issue where valid cached tokens are ignored during backoff after a failed background refresh, and a potential integer overflow when calculating expirationTimeMillis from large exp claims.
| synchronized (lock) { | ||
| if (readState == ReadState.BACKOFF && now >= nextAttemptTimeMillis) { | ||
| readState = ReadState.IDLE; | ||
| } | ||
|
|
||
| if (readState == ReadState.BACKOFF) { | ||
| failStatus = lastReadFailureStatus != null ? lastReadFailureStatus : Status.UNAVAILABLE; | ||
| } else { | ||
| boolean hasValidCache = cachedToken != null && now < expirationTimeMillis; | ||
| boolean expiringSoon = hasValidCache && (expirationTimeMillis - now <= 60000); | ||
|
|
||
| if (hasValidCache) { | ||
| tokenToApply = new TokenInfo(cachedToken, expirationTimeMillis); | ||
| if (expiringSoon && readState == ReadState.IDLE) { | ||
| readState = ReadState.READING; | ||
| triggerRead = true; | ||
| } | ||
| } else { | ||
| if (readState == ReadState.IDLE) { | ||
| readState = ReadState.READING; | ||
| triggerRead = true; | ||
| } | ||
| queuedAppliers.add(applier); | ||
| } | ||
| } | ||
| } |
There was a problem hiding this comment.
Critical Bug: Valid cached token is ignored and requests fail during backoff
When a background refresh fails (e.g., due to a transient file read error), the credential enters the BACKOFF state. Under the current implementation, any request made during the backoff window will fail immediately because readState == ReadState.BACKOFF is checked first and causes an immediate failure, completely ignoring the fact that a perfectly valid cached token still exists in cachedToken.
This can cause severe availability issues where healthy, unexpired tokens are not used simply because a background refresh attempt failed.
We should check if a valid cache exists first. If it does, we should always use it and only trigger/skip background refreshes based on the backoff state. If no valid cache exists, only then should we fail-fast during the backoff window.
synchronized (lock) {
boolean hasValidCache = cachedToken != null && now < expirationTimeMillis;
boolean expiringSoon = hasValidCache && (expirationTimeMillis - now <= 60000);
if (hasValidCache) {
tokenToApply = new TokenInfo(cachedToken, expirationTimeMillis);
if (readState == ReadState.BACKOFF && now >= nextAttemptTimeMillis) {
readState = ReadState.IDLE;
}
if (expiringSoon && readState == ReadState.IDLE) {
readState = ReadState.READING;
triggerRead = true;
}
} else {
if (readState == ReadState.BACKOFF && now >= nextAttemptTimeMillis) {
readState = ReadState.IDLE;
}
if (readState == ReadState.BACKOFF) {
failStatus = lastReadFailureStatus != null ? lastReadFailureStatus : Status.UNAVAILABLE;
} else {
if (readState == ReadState.IDLE) {
readState = ReadState.READING;
triggerRead = true;
}
queuedAppliers.add(applier);
}
}
}| long expSeconds = expElement.getAsLong(); | ||
| if (expSeconds <= 0) { | ||
| throw new IllegalArgumentException("Invalid 'exp' claim value: " + expSeconds); | ||
| } | ||
| long expirationTimeMillis = (expSeconds - 30) * 1000; | ||
| return new TokenInfo(token, expirationTimeMillis); |
There was a problem hiding this comment.
Medium Bug: Potential integer overflow when calculating expirationTimeMillis
If the exp claim in the JWT is extremely large (e.g., Long.MAX_VALUE or close to it), multiplying (expSeconds - 30) by 1000 can overflow long and result in a negative value. This would cause the token to be treated as expired immediately, leading to infinite reload loops or authentication failures.
We should add a defensive check to prevent overflow by capping the expiration time at Long.MAX_VALUE if expSeconds is too large.
| long expSeconds = expElement.getAsLong(); | |
| if (expSeconds <= 0) { | |
| throw new IllegalArgumentException("Invalid 'exp' claim value: " + expSeconds); | |
| } | |
| long expirationTimeMillis = (expSeconds - 30) * 1000; | |
| return new TokenInfo(token, expirationTimeMillis); | |
| long expSeconds = expElement.getAsLong(); | |
| if (expSeconds <= 0) { | |
| throw new IllegalArgumentException("Invalid 'exp' claim value: " + expSeconds); | |
| } | |
| long expirationTimeMillis; | |
| if (expSeconds > Long.MAX_VALUE / 1000) { | |
| expirationTimeMillis = Long.MAX_VALUE; | |
| } else { | |
| expirationTimeMillis = (expSeconds - 30) * 1000; | |
| } | |
| return new TokenInfo(token, expirationTimeMillis); |
Implements gRFC A97 to support file-based JSON Web Token (JWT) Call Credentials for xDS-enabled clients. Implemented using an experimental coding agent.
Summary of Changes