OTLP gRPC: isRetryable ignores grpc-status in trailers, so UNAVAILABLE is never retried

[kevinburkesegment]

Describe the bug

OkHttpGrpcSender reads grpc-status in two places, and they disagree:

• grpcStatus() (OkHttpGrpcSender.java:291) reads the header, then falls back to trailers
• isRetryable() (OkHttpGrpcSender.java:368) reads the header only, and returns false when it is absent

So when the status arrives in trailers, the exporter reports Server is UNAVAILABLE to the user but never retries — the batch is dropped after a single attempt, with the default
maxAttempts=5 in effect.

The existing comment says:

// We don't check trailers for retry since retryable error codes always come with response
// headers, not trailers, in practice.

Per PROTOCOL-HTTP2.md, trailers are the normal location, not the exception:

Status must be sent in Trailers even if the status code is OK.

Most responses are expected to have both headers and trailers but Trailers-Only is permitted for calls that produce an immediate error.

isRetryable handles only the Trailers-Only case. This is observable behind a proxy: Envoy sends Trailers-Only for failures before upstream response headers are forwarded, and real trailers once the upstream has already sent headers and then fails. The same backend outage is therefore retried or not depending on when it fails.

Steps to reproduce

Stand up a local h2c server returning grpc status 14, first in the headers, then in the trailers.

@Test
void isRetryable_grpcStatusInTrailers() {
    Response response =
        new Response.Builder()
            .request(new Request.Builder().url("http://localhost/").build())
            .protocol(Protocol.HTTP_2)
            .code(200)
            .message("OK")
            .header("content-type", "application/grpc")
            .body(ResponseBody.create("", TEXT_PLAIN))
            .trailers(() -> Headers.of(GRPC_STATUS, "14")) // UNAVAILABLE
            .build();

    // grpcStatus(response) already reports UNAVAILABLE from these same trailers.
    assertThat(OkHttpGrpcSender.isRetryable(response)).isTrue(); // actual: false
}

Run twice:

--- grpc-status:14 in HEADERS (Trailers-Only) ---
Attempt 0..3 returned retryable response                         FAILURE, 4 attempts
--- grpc-status:14 in TRAILERS ---
Attempt 0 returned non-retryable response
[SEVERE] Failed to export logs. Server is UNAVAILABLE. Make sure your collector is
         running and reachable from this network.                FAILURE, 1 attempt

What did you expect to see?

The request should be retried if UNAVAILABLE is in the http trailer

For comparison, opentelemetry-go is structurally immune because it decides on the resolved gRPC status rather than on HTTP headers:

  func retryable(err error) (bool, time.Duration) {
        s := status.Convert(err)
        return retryableGRPCStatus(s)
  }

What did you see instead?

Log level severe, no retry.

What version and what artifacts are you using?

Reproduced on 1.60.1, 1.64.0, and main (711e482)

Environment

Should not matter

Additional context

Here's a possible patch:

public static boolean isRetryable(Response response) {
    String grpcStatus = response.header(GRPC_STATUS);
    if (grpcStatus == null) {
      try {
        grpcStatus = response.trailers().get(GRPC_STATUS);   // mirrors grpcStatus()
      } catch (IOException e) {
        return false;
      }
    }
    return grpcStatus != null && RetryUtil.retryableGrpcStatusCodes().contains(grpcStatus);
  }

If you want to keep the 4MiB guard in handleResponse, it gets a bit more complicated, but it's still doable.

I'm happy to implement and submit, let me know.