I installed the continuous integration service Woodpecker, to replace DroneCI, which the company that bought it decided to bury. As Woodpecker is a fork of the latest free version of Drone, its use is broadly similar.
However, the teams have taken different directions on certain aspects, and communication with agents/runners, which used to be via websockets, is now carried out in Woodpecker using the GRPC protocol.
The solution proposed by the Woodpecker documentation is to use 2 domains: one will be used for the web interface and the REST API, the second will be used for GRPC. Is this really necessary?
nginx configuration using 2 domains
Woodpecker exposes the web interface and GRPC on two different ports: 8000 and 9000 respectively.
The simplest approach to exposing these two services on the Internet is to use two separate domains. Here’s what our reverse-proxy configuration might look like:
server {
listen 80;
server_name woodpecker.example.com;
location / {
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host $http_host;
proxy_pass http://127.0.0.1:8000;
proxy_redirect off;
proxy_http_version 1.1;
proxy_buffering off;
chunked_transfer_encoding off;
}
}
server {
listen 80 http2;
server_name woodpeckeragent.example.com;
location / {
grpc_pass grpc://127.0.0.1:9000;
}
}
Here we’re using nginx’s GRPC module, and in particular the grpc_pass directive.
This directive is similar to the proxy_pass directive: it will forward to port 9000 on the local machine all GRPC packets arriving on the woodpeckeragent.example.com.
On the other hand, I prefer to avoid declaring 2 domains, obtaining 2 certificates, etc. for 1 and the same service, with a domain that you tend to forget and neglect. So let’s see if we can’t do better.
The GRPC protocol
Without going into too much detail, GRPC is a protocol close to HTTP/2.
As such, many reverse-proxies are capable of transmitting GRPC requests.
This is the case of Caddy, a brief example of which is given, Traefik.
But nginx also supports the transmission of GRPC requests.
When a GRPC request is received by the reverse-proxy, it sees an HTTP/2 request similar to:
POST /pkg.Service/Function HTTP/2.0
Host: grpc.example.com
User-Agent: grpc-go/1.21.0
[...]
At first glance, there’s nothing confusing for a web server. There must be something clever we can do to use these similarities to our advantage.
Combine HTTP requests and GRPC calls on the same domain
The path for GPRC requests is fixed, and depends on the protobuf file describing calls and structures.
Each Service is declared within a package (pkg), then Functions complete the path.
All calls are POSTs.
So we need to extract the various HTTP routes used by each protobuf service. Let’s take a look at the following file for Woodpecker:
https://github.com/woodpecker-ci/woodpecker/blob/main/pipeline/rpc/proto/woodpecker.proto
[...]
package proto;
[...]
service Woodpecker {
rpc Version (Empty) returns (VersionResponse) {}
rpc Next (NextRequest) returns (NextResponse) {}
rpc Init (InitRequest) returns (Empty) {}
rpc Wait (WaitRequest) returns (Empty) {}
rpc Done (DoneRequest) returns (Empty) {}
rpc Extend (ExtendRequest) returns (Empty) {}
rpc Update (UpdateRequest) returns (Empty) {}
rpc Log (LogRequest) returns (Empty) {}
rpc RegisterAgent (RegisterAgentRequest) returns (RegisterAgentResponse) {}
rpc ReportHealth (ReportHealthRequest) returns (Empty) {}
}
[...]
service WoodpeckerAuth {
rpc Auth (AuthRequest) returns (AuthResponse) {}
}
[...]
From this descriptive file, we have now determined all the routes that can be used by all clients using this version of the service.
The proto package contains 2 services: Woodpecker and WoodpeckerAuth.
This gives us 2 root routes:
/proto.Woodpecker//proto.WoodpeckerAuth/
Behind each, we find the functions described by each rpc line.
For example:
/proto.Woodpecker/Version/proto.Woodpecker/Done/proto.Woodpecker/Log/proto.WoodpeckerAuth/Auth- …
nginx configuration
So here’s what our nginx configuration might look like, using a single domain:
server {
listen 80;
http2 on;
server_name woodpecker.example.com;
location / {
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host $http_host;
proxy_pass http://127.0.0.1:8000;
proxy_redirect off;
proxy_http_version 1.1;
proxy_buffering off;
chunked_transfer_encoding off;
}
location /proto.Woodpecker/ {
grpc_pass grpc://127.0.0.1:9000;
}
location /proto.WoodpeckerAuth/ {
grpc_pass grpc://127.0.0.1:9000;
}
}
Note that, of course, this only works if HTTP/2 is enabled by means of the http2 directive.
There’s no need to specify functions individually, as nginx will perform the routing using only the package and service name.
Timeout on the Next function
Woodpecker declares an Next function that waits for information about the next job to be performed.
This waiting time is generally quite variable, and can be quite long if your IC is not constantly in use.
nginx will take the initiative, by default, to stop any request that has been inactive for more than 60 seconds.
This causes a large number of reconnections for the Woodpecker agent, visible in the agent logs:
{"level":"error","error":"rpc error: code = Unavailable desc = unexpected HTTP status code received from server: 504 (Gateway Timeout); transport: received unexpected content-type \"text/html\"","time":"2023-10-27T11:28:51Z","message":"grpc error: wait(): code: Unavailable: rpc error: code = Unavailable desc = unexpected HTTP status code received from server: 504 (Gateway Timeout); transport: received unexpected content-type \"text/html\""}
The agent will reconnect itself, but we can reduce the occurrence of this reconnection (whose main benefit is to do nothing until the server wakes us up), by adding this line to our configuration nginx:
location /proto.Woodpecker/Next {
grpc_pass grpc://127.0.0.1:9008;
grpc_read_timeout 3600s;
}
These few lines tell nginx not to disconnect inactive connections before 1 hour of inactivity, only for the Next function of the protocol.
Filter IPs that can access the GRPC service
In addition to the grpc_pass directive, you can of course use all the other directives you’re used to.
For example, we might want to filter the IPs that have access to GRPC.
We could do this with the allow and deny directives:
location /proto.Woodpecker/ {
allow 192.168.0.0/24;
deny all;
grpc_pass grpc://127.0.0.1:9000;
}
Since only agents use the GRPC protocol, it would be perfectly legitimate to implement such a rule.
Unifying HTTP requests and GRPC calls under a single domain offers many advantages. Not only does this approach simplify reverse-proxy configuration, it also enables more centralized, streamlined management of the service.
By the way, GRPC and protobuf are an excellent solution for transmitting structured data over the network, so don’t hesitate to take a look.