The API Pods are healthy. A direct request to a Pod succeeds. A request through the Service times out. Restarting every Pod would destroy useful evidence without explaining the failure.
A Service gives clients a stable way to discover a changing set of backends. Understanding that separation turns a mysterious outage into a sequence of testable handoffs.
Start with the traffic contract
For an ordinary selector-based ClusterIP Service, the useful model is client → DNS → Service address and port → data plane → eligible endpoint → application listener. DNS resolves a name; it does not establish application health. The virtual IP represents a routing abstraction; it is not a process listening inside a server.
Controllers watch labels and publish endpoint information in EndpointSlices. A node's Service implementation, commonly kube-proxy or an alternative provided by the networking stack, programs forwarding. The exact packet path depends on the implementation. Avoid assuming every cluster uses the same iptables rules.
Three names that must agree
apiVersion: v1
kind: Service
metadata:
name: checkout
namespace: shop
spec:
selector:
app: checkout
ports:
- name: http
port: 80
targetPort: 8080
This declares a client-facing port of 80 and a backend destination port of 8080. It selects Pods labeled app=checkout in the same namespace. It does not cause an application to listen on 8080. A containerPort declaration documents a port; the application must actually bind it, usually on an interface reachable from other Pods rather than loopback alone.
The name checkout.shop.svc.cluster.local is a typical fully qualified name when cluster.local is the cluster domain. Short-name resolution depends on the client's namespace and DNS search configuration.
Endpoint existence and endpoint eligibility
A selector can match zero Pods because of a spelling error or a release label change. A matching Pod can exist but be unready. Inspect EndpointSlice conditions rather than equating an endpoint address with a backend receiving ordinary traffic.
Readiness normally determines eligibility. Special configurations such as publishNotReadyAddresses change this behavior. Terminating endpoint handling and traffic policies add nuance. The lab deliberately models the common non-terminating, readiness-filtered case with no session affinity.
Experiment: remove a backend without changing the address
Use the routing lab above. Make a Pod unready, change the selector, or break the target port. Watch three separate facts: whether DNS resolves, whether an eligible backend exists, and whether the application can accept the forwarded request. Then send requests and compare their outcomes.
The simulated round-robin choice is illustrative. Kubernetes does not promise equal per-request distribution. Connection reuse, affinity, topology, and implementation details can concentrate traffic on a subset of Pods.
A production debugging sequence
kubectl -n shop get svc checkout -o yaml
kubectl -n shop get pods -l app=checkout --show-labels
kubectl -n shop get endpointslices -l kubernetes.io/service-name=checkout -o yaml
kubectl -n shop describe pod checkout-example
First record the failing client, destination, port, protocol, and time. Test DNS from the affected network context. Compare a request to the Service with a request to an endpoint from the same client context. A Pod port-forward uses a different path and cannot prove that normal Pod-to-Pod networking works.
No endpoints? Check namespace, selector, labels, readiness, and controller state. Endpoints exist but direct traffic fails? Check the listener, target port, Pod network, and applicable policies. Direct traffic works but Service traffic fails? Inspect the Service data plane and traffic policies. External access fails while internal access works? Continue outward through Gateway or Ingress, load balancer, firewall, DNS, and TLS.
Commands are observations, not a checklist to run indiscriminately. A timeout and a connection refusal are different evidence. Neither alone identifies the responsible component.
Service types change the contract
ClusterIP is the usual internal virtual destination. NodePort exposes a port on nodes subject to networking and firewall configuration. LoadBalancer requests an external load balancer from an implementation; it does not guarantee instant provisioning. A headless Service, with clusterIP: None, exposes endpoint discovery without the usual virtual IP. ExternalName returns a DNS alias and has no selector-based proxying to Pods.
Long-lived connections deserve special care. Adding replicas does not move existing connections. Removing a backend does not make every in-flight request succeed. Graceful shutdown, readiness transitions, connection draining, and client retry behavior must cooperate.
Keep the boundary visible
The address is stable. Its backing set is dynamic. Availability depends on discovery, endpoint eligibility, forwarding, and the application agreeing about the same traffic contract.
Sources and scope
This guide's packet path is a conceptual model. The lab uses hypothetical Pods and deterministic request selection, not a real cluster.






