Can't DNAT After DNAT?
When dealing with Kubernetes and network configurations, we often encounter DNAT (Destination NAT) mechanisms. Whether it's "rewriting Service Virtual IP to Pod IP" or "forwarding external access to specific nodes or Pods" - DNAT is working behind the scenes in these cases. However, this raises an interesting question: "Can you rewrite a DNAT-ed destination with another DNAT?" Many people encounter this question during various cases and troubleshooting. To cut to the chase, the answer is generally "no" or "it's difficult." This article explains the mechanism and reasons behind this. 1. What is DNAT? Let's first review what DNAT is. DNAT (Destination NAT) is a technology that rewrites destination information (IP address and port) of incoming packets. In Linux's netfilter/iptables, it's primarily configured in the PREROUTING chain of the nat table. Example: "Forward external traffic coming to Node's port 80 to Pod's port 8080" In PREROUTING, specify -j DNAT --to-destination :8080 for packets matching -d --dport 80 In Kubernetes, this DNAT is automatically configured on each node, implementing a mechanism to distribute Service (ClusterIP) destinations to Pod IPs. 2. Why Can't You "DNAT After DNAT"? In conclusion, double DNAT is basically impossible with iptables. The main reasons are: iptables NAT assumes one transformation per connection iptables NAT works with connection tracking (conntrack) to perform NAT on a per-packet-flow basis. Once DNAT is applied to the initial packet, subsequent packets in the same connection are treated as "already transformed" and generally won't undergo DNAT again. Routing occurs after DNAT, so packets don't return to PREROUTING DNAT is typically applied in PREROUTING. After the destination IP is rewritten here, the Linux kernel's routing table determines the packet's destination. Packets flowing in the same direction won't re-enter PREROUTING, making it impossible to "DNAT after DNAT." Double DNAT is possible with special configurations, but it's not common Theoretically, you can perform another DNAT after packets are delivered to a different host. However, multi-stage configurations like "PREROUTING → DNAT → DNAT again on the same host" aren't intended in standard iptables. 3. DNAT in Kubernetes Services In Kubernetes, packets destined for Services (ClusterIP) are forwarded by DNATing them to Pod IPs. Specifically, the following steps are executed in KUBE-SERVICES and KUBE-POSTROUTING chains created on the Node: DNAT in PREROUTING Packets addressed to Service ClusterIP are rewritten to the selected Pod IP:Port. Routing table determines Pod destination After rewriting, routing occurs either internally to the Node where the Pod is running or to another Node. Naturally, once the destination is determined by one DNAT, further DNAT operations aren't expected. This directly relates to why "DNAT after DNAT" is difficult. 4. Common Questions and Troubleshooting 4.1 "External Load Balancer → NodePort → Pod, More DNAT?" In Kubernetes, when external load balancers route traffic to NodePort and then to Pods, DNAT occurs. However, this is essentially one flow of "external LB → NodePort → DNAT → Pod" tracked as a single continuous connection, not double DNAT. 4.2 "Is DNAT Performed Again When Forwarding from One Service to Another?" When requesting from Service A to Service B on the same Node, DNAT might occur again as the packets are treated as a separate connection. However, this is just DNATing "as a completely separate connection." 5. Hints for When You Still Need Double DNAT If you need to implement "DNAT after DNAT" for some requirements, consider these approaches: Insert a different host or device After DNAT is applied, you can apply new DNAT when packets are delivered to a different host. Separate NAT in OUTPUT or POSTROUTING chains You might create complex configurations like DNAT in one direction and SNAT in the return direction. However, this is hard to manage and not recommended. Generally, the disadvantages of network complexity outweigh the benefits, so it's recommended to avoid "multi-stage DNAT" during design if possible. Summary Reasons why you can't DNAT after DNAT iptables NAT works with connection tracking (conntrack) and "won't transform an already transformed connection" After DNAT, routing occurs, so the same packet doesn't return to PREROUTING Kubernetes Services and NodePort work the same way Both finish after one DNAT rewrites the destination to Pod IP If double DNAT is absolutely necessary Special configurations like routing through different hosts are needed but aren't common Understanding iptables' design principle of "one NAT per connection" helps with troubleshooting Kubernetes and network issues. When designing systems, it's recommended to keep flows simple to avoid complications from
When dealing with Kubernetes and network configurations, we often encounter DNAT (Destination NAT) mechanisms. Whether it's "rewriting Service Virtual IP to Pod IP" or "forwarding external access to specific nodes or Pods" - DNAT is working behind the scenes in these cases.
However, this raises an interesting question: "Can you rewrite a DNAT-ed destination with another DNAT?" Many people encounter this question during various cases and troubleshooting. To cut to the chase, the answer is generally "no" or "it's difficult." This article explains the mechanism and reasons behind this.
1. What is DNAT?
Let's first review what DNAT is. DNAT (Destination NAT) is a technology that rewrites destination information (IP address and port) of incoming packets. In Linux's netfilter/iptables, it's primarily configured in the PREROUTING chain of the nat table.
- Example: "Forward external traffic coming to Node's port 80 to Pod's port 8080"
- In PREROUTING, specify -j DNAT --to-destination :8080 for packets matching -d --dport 80
In Kubernetes, this DNAT is automatically configured on each node, implementing a mechanism to distribute Service (ClusterIP) destinations to Pod IPs.
2. Why Can't You "DNAT After DNAT"?
In conclusion, double DNAT is basically impossible with iptables. The main reasons are:
-
iptables NAT assumes one transformation per connection
- iptables NAT works with connection tracking (conntrack) to perform NAT on a per-packet-flow basis. Once DNAT is applied to the initial packet, subsequent packets in the same connection are treated as "already transformed" and generally won't undergo DNAT again.
-
Routing occurs after DNAT, so packets don't return to PREROUTING
- DNAT is typically applied in PREROUTING. After the destination IP is rewritten here, the Linux kernel's routing table determines the packet's destination. Packets flowing in the same direction won't re-enter PREROUTING, making it impossible to "DNAT after DNAT."
-
Double DNAT is possible with special configurations, but it's not common
- Theoretically, you can perform another DNAT after packets are delivered to a different host. However, multi-stage configurations like "PREROUTING → DNAT → DNAT again on the same host" aren't intended in standard iptables.
3. DNAT in Kubernetes Services
In Kubernetes, packets destined for Services (ClusterIP) are forwarded by DNATing them to Pod IPs. Specifically, the following steps are executed in KUBE-SERVICES and KUBE-POSTROUTING chains created on the Node:
-
DNAT in PREROUTING
- Packets addressed to Service ClusterIP are rewritten to the selected Pod IP:Port.
-
Routing table determines Pod destination
- After rewriting, routing occurs either internally to the Node where the Pod is running or to another Node.
Naturally, once the destination is determined by one DNAT, further DNAT operations aren't expected. This directly relates to why "DNAT after DNAT" is difficult.
4. Common Questions and Troubleshooting
4.1 "External Load Balancer → NodePort → Pod, More DNAT?"
In Kubernetes, when external load balancers route traffic to NodePort and then to Pods, DNAT occurs. However, this is essentially one flow of "external LB → NodePort → DNAT → Pod" tracked as a single continuous connection, not double DNAT.
4.2 "Is DNAT Performed Again When Forwarding from One Service to Another?"
When requesting from Service A to Service B on the same Node, DNAT might occur again as the packets are treated as a separate connection. However, this is just DNATing "as a completely separate connection."
5. Hints for When You Still Need Double DNAT
If you need to implement "DNAT after DNAT" for some requirements, consider these approaches:
-
Insert a different host or device
- After DNAT is applied, you can apply new DNAT when packets are delivered to a different host.
-
Separate NAT in OUTPUT or POSTROUTING chains
- You might create complex configurations like DNAT in one direction and SNAT in the return direction. However, this is hard to manage and not recommended.
Generally, the disadvantages of network complexity outweigh the benefits, so it's recommended to avoid "multi-stage DNAT" during design if possible.
Summary
-
Reasons why you can't DNAT after DNAT
- iptables NAT works with connection tracking (conntrack) and "won't transform an already transformed connection"
- After DNAT, routing occurs, so the same packet doesn't return to PREROUTING
-
Kubernetes Services and NodePort work the same way
- Both finish after one DNAT rewrites the destination to Pod IP
-
If double DNAT is absolutely necessary
- Special configurations like routing through different hosts are needed but aren't common
Understanding iptables' design principle of "one NAT per connection" helps with troubleshooting Kubernetes and network issues. When designing systems, it's recommended to keep flows simple to avoid complications from double DNAT.
