【问题记录】Istio与Prometheus冲突问题，“GET /metrics HTTP/1.1“ 503 UF upstream_reset_before_response_started{rem

问题

使用istio官网提供的安装方式，默认会安装loki、kiali、prometheus、grafana，如果自己再搭建一个prometheus，会出现冲突

最开始我发现问题是查看kiali Graph面板出现大量请求错误，查看被请求的日志，发现很多下面的字段：

"GET /metrics HTTP/1.1" 503 UF upstream_reset_before_response_started{remote_connection_failure,delayed_connect_error:_111} - "delayed_connect_error:_111" 0 152 0 - "-" "Prometheus/2.47.2" "c74e755d-6d2f-9131-bf7d-39018dcd576b" "10.0.9.69:30016" "10.0.9.69:30016" inbound|30016|| - 10.0.9.69:30016 10.0.16.108:52286 - default

127.0.0.6 - - [26/Mar/2024:06:10:50 +0000] "GET /metrics HTTP/1.1" 200 537 "-" "Prometheus/2.47.2" "-"
127.0.0.6 - - [26/Mar/2024:06:11:00 +0000] "GET /metrics HTTP/1.1" 200 537 "-" "Prometheus/2.47.2" "-"
127.0.0.6 - - [26/Mar/2024:06:11:10 +0000] "GET /metrics HTTP/1.1" 200 537 "-" "Prometheus/2.47.2" "-"
127.0.0.6 - - [26/Mar/2024:06:11:20 +0000] "GET /metrics HTTP/1.1" 200 537 "-" "Prometheus/2.47.2" "-"
127.0.0.6 - - [26/Mar/2024:06:11:30 +0000] "GET /metrics HTTP/1.1" 200 537 "-" "Prometheus/2.47.2" "-"
127.0.0.6 - - [26/Mar/2024:06:11:40 +0000] "GET /metrics HTTP/1.1" 200 537 "-" "Prometheus/2.47.2" "-"
127.0.0.6 - - [26/Mar/2024:06:11:50 +0000] "GET /metrics HTTP/1.1" 200 537 "-" "Prometheus/2.47.2" "-"
127.0.0.6 - - [26/Mar/2024:06:12:00 +0000] "GET /metrics HTTP/1.1" 200 537 "-" "Prometheus/2.47.2" "-"
127.0.0.6 - - [26/Mar/2024:06:12:10 +0000] "GET /metrics HTTP/1.1" 200 537 "-" "Prometheus/2.47.2" "-"
127.0.0.6 - - [26/Mar/2024:06:12:20 +0000] "GET /metrics HTTP/1.1" 200 537 "-" "Prometheus/2.47.2" "-"
127.0.0.6 - - [26/Mar/2024:06:12:30 +0000] "GET /metrics HTTP/1.1" 200 537 "-" "Prometheus/2.47.2" "-"
127.0.0.6 - - [26/Mar/2024:06:12:40 +0000] "GET /metrics HTTP/1.1" 200 537 "-" "Prometheus/2.47.2" "-"
127.0.0.6 - - [26/Mar/2024:06:12:50 +0000] "GET /metrics HTTP/1.1" 200 537 "-" "Prometheus/2.47.2" "-"
127.0.0.6 - - [26/Mar/2024:06:13:00 +0000] "GET /metrics HTTP/1.1" 200 537 "-" "Prometheus/2.47.2" "-"
127.0.0.6 - - [26/Mar/2024:06:13:10 +0000] "GET /metrics HTTP/1.1" 200 537 "-" "Prometheus/2.47.2" "-"

解决

删除了添加这种pod的注释，改为手动添加

      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/path: /metrics
        prometheus.io/port: "3000"

此时，istio代理的服务没有出现以上错误请求，具体原因还在查找中