GRPC Health Checks in Kubernetes
The gRPC framework is becoming the de facto standard for backend services. Services no longer need to implement custom REST support given the transcoding provided by cloud platforms. These transcoding appliances expose REST endpoints that forward requests to the deployed gRPC service. If there is no business requirement for implementing custom REST endpoints, then why would we do so just to support liveness health checks?
Support in Kubernetes
Support for using gRPC for health checks has lagged the frameworks adoption. This is to be expected. The old way (before K8s 1.23) was running an independent health probe. Before the release of K8s 1.23 an independent health probe was required to query the health of gRPC services. The gRPC liveness probe was then introduced with the release of K8s 1.24. This version also enables the required GRPCContainerProbe Feature Gate by default. These are quick advances in gRPC support. What does it really take to deploy a service that uses gRPC health checks? What are the minimum requirements?
Example service
The simplest K8s YAML configuration to demonstrate liveness health checks might be less than the following. It is important to consider what we are health checking. The following example configures a health check upon a service which is overlayed upon a deployment with one container.
apiVersion: apps/v1
kind: Deployment
metadata:
name: example-deploy
namespace: default
spec:
replicas: 1
selector:
matchLabels:
app: example
template:
metadata:
labels:
app: example
spec:
containers:
- name: example
image: example:v0
imagePullPolicy: Never
ports:
- containerPort: 8000
livenessProbe:
grpc:
port: 8000
initialDelaySeconds: 5
periodSeconds: 10
---
apiVersion: v1
kind: Service
metadata:
name: example-service
spec:
# type: NodePort
selector:
app: example
ports:
- protocol: TCP
port: 8000
Let's take a closer look at what this basic example creates in the K8s cluster. The most important tidbit is the liveness probe on the container:
livenessProbe:
grpc:
port: 8000
The full YAML configuration will create a deployment with one container that has a service on port 8000. The service is implemented in a Docker image tagged example:v0. That Docker image will be deployed as a single pod. In order to meet the requirements for the gRPC livenessProbe, the service implemented and deployed in the Docker image must implement that health check.
Implementing the health service
The health check contract that K8s expects is the GRPC Health Checking Protocol. There are implementations available for most languages. This main.go
shows the full registration of the gRPC health service that always returns success.
package main
import (
"flag"
"fmt"
"log"
"net"
"google.golang.org/grpc"
"google.golang.org/grpc/health"
"google.golang.org/grpc/health/grpc_health_v1"
"google.golang.org/grpc/reflection"
)
func main() {
port := flag.Int("port", 8000, "The server port")
host := flag.String("host", "", "The server host, perhaps 'localhost'")
flag.Parse()
s := grpc.NewServer()
defer s.Stop()
hs := health.NewServer() // will default to respond with SERVING
grpc_health_v1.RegisterHealthServer(s, hs) // registration
// register your own services
reflection.Register(s)
address := fmt.Sprintf("%s:%d", *host, *port)
log.Default().Printf("starting service on %s", address)
listener, err := net.Listen("tcp", address)
if err != nil {
panic(err)
}
err = s.Serve(listener)
if err != nil {
panic(err)
}
}
The key piece here is that the default implementation of the health service is registered.
hs := health.NewServer() // will default to respond with SERVING
grpc_health_v1.RegisterHealthServer(s, hs) // registration
All that's left is to build a Docker image (docker build -t example:v0 .
) and then deploy it to K8s with the above YAML file: kubectl apply -f service.yaml
.
Did it work? Is the health endpoint being called? Simply confirming that the pod has not been restarted (kubectl get pods
) doesn't really tell us the health check is passing. Maybe the health check isn't even being called. Some adjustments could be made to the provided health server, however, implementing the service is simple. Here is a minimal example implementation.
grpc_health_v1.RegisterHealthServer(s, &Health{}) // replaces default registration
...
type Health struct {
grpc_health_v1.UnimplementedHealthServer
}
func (h Health) Check(context.Context, *grpc_health_v1.HealthCheckRequest) (*grpc_health_v1.HealthCheckResponse, error) {
log.Default().Println("serving health")
return &grpc_health_v1.HealthCheckResponse{Status: grpc_health_v1.HealthCheckResponse_SERVING}, nil
}
This example changes the grpc_health_v1 registration to a custom health server that will log and return SERVING. The steps below can be used to deploy the updated service so that it logs (kubectl logs deploy/example-deploy
) a new line every 10 seconds. That is the period configured in the YAML above.
- Build a new Docker image:
docker build -t example:v1
. - Update the service.yaml to reference the new version of the Docker image: (
image: example:v1
). - Deploy to K8s: (
kubectl apply -f service.yaml
).
This confirms the health check is being called. Should we have a little fun with it? It's simple to put the deployed service pod into crash loop backoff. Just change the returned health status to grpc_health_v1.HealthCheckResponse_NOT_SERVING
.
Conclusion
Over a short period of time K8s support for gRPC health checks has moved from full custom solutions to native support. Kubernetes 1.25 (and probably above) requires no extra components to health check your service through the gRPC framework. Be sure to locate information that is specific to the Kubernetes version you are using when reading documentation and articles. Build from the basic example with verification at each step to review your progress. The subsequent steps might be deprecated even if not documented as such.