Kubernetes

Istio Upstream Connection Overflow

  • Kubernetes
  • K8S
  • Openshift
  • Istio

Background

I found some failed request 503 error for the microservice deployment in Openshift 4.5 with Service Mesh v1. The error shown is upstream connect error or disconnect/reset before headers. reset reason: overflow.

Problem

I found the solution during the search and refer to the documentation. It is because Service Mesh v1 is using Istio 1.3. And the Connection poll is set to 1024 for http1MaxPendingRequests, http2MaxRequests, etc. It is different from version 1.8 where it is default to 2^32-1. Therefore, it is required to set it differently.

Solution

This is not really a good solution I believe. and it require sometime to investigate, but you can increase those settings with destination rules.

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: sample-destinationrule
spec:
  host: sample
  trafficPolicy:
    loadBalancer:
      simple: RANDOM
    connectionPool:
      http:
        http1MaxPendingRequests: 10240
        http2MaxRequests: 10240
  subsets:
  - name: v1
    labels:
      version: "v1"

Comments