Nginx reverse proxy solve no avaliable
14 May 2019Background
We use a main server reverse proxy all the request to other backend server. So, it means that the client access the main server, and nginx reverse proxy the request to the other serever.
So, we use subdomain to dynamically resolve the address so it is working well when server ip changes.
Problem
Today, the nginx is not working as expected. It redirect all request to 502 page. And we found the error log.
2019/05/14 08:41:15 [error] 688#688: *4 connect() failed (110: Connection timed out) while connecting to upstream, client: 113.*.*.*, server: furry.top, request: "GET / HTTP/2.0", upstream: "https://116.*.*.*:443/", host: "furry.top", referrer: "https://furry.top/"
2019/05/14 08:41:15 [error] 688#688: *4 no live upstreams while connecting to upstream, client: 113.*.*.*, server: furry.top, request: "GET / HTTP/2.0", upstream: "https://https/", host: "furry.top", referrer: "https://furry.top/"
It is quite weird because when we reload the server it starts working. And then stop working after 10 seconds. We could not found why this happend.
Attempts
I assume that it is because of the unstable connection. So, I check append max_fails=0
config to ensure upstream
does not fail. However, it gets worse. Nginx basically try to connect to backend continuously and not reply to any connection. However, when I curl
backend address, it has successfully responsed. So nothing wrong with the connection but nginx some how cannot connect to the backend.
I have no idea why it happend. Then I check the connection with netstat -ap
. One line draw my attention.
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 1 main-server.a:36298 166.*.*.*:http SYN_SENT 10128/nginx: worker
Since there is no respond from the address, it did not connect to correct backend address. But, why it only works for small period of time? Does it mean that the dns server did not work?
So I check in the configuration file again. I found resolver 114.114.114.114
, this changes the DNS server. So, to verify the address. I use dig
command, dig @114.114.114.114 furry.top
and the result is that 114.114.114.114
is not working. So, I replace the DNS with 8.8.8.8
and the problem solved.
So, that’s why nginx fail to connect to backend but I cannot find any log indicate that the DNS failed. It is because I did not check the default logging file.
[error] 2439#2439: upstream-dynamic-servers: 'furry.top' could not be resolved (110: Operation timed out)
The answer was there.