Skip to content

Understanding Limitations and Advanced Features of Rumble Cloud Kubernetes Clusters

When deploying Kubernetes clusters on Rumble Cloud, it's important to understand how certain networking features can interact, especially when using Floating IPs in combination with Load Balancers. Improper configuration can lead to subtle and hard-to-debug issues—particularly IP address conflicts caused by overlapping address translation mechanisms.

The problem: IP conflict between floating IPs and load balancers

Support investigations uncovered a potential source of networking problems in Rumble Cloud Kubernetes environments: assigning Floating IPs directly to cluster nodes while simultaneously deploying LoadBalancer-type services in Kubernetes.

Both features involve address translation: - Floating IPs translate public IP traffic to the node's internal IP. - Load balancers perform NAT to direct incoming traffic to service endpoints (pods or nodes).

When both systems operate at once, conflicts in NAT rules can arise. This may prevent the load balancers from functioning properly, manifesting as failed health checks or unreachable services.

To prevent or mitigate these issues, the following best practices are recommended:

Avoid assigning floating IPs to nodes initially

The Rumble Cloud provided cluster templates do not assign floating ip's directly to nodes by default. However, it is possible via the command line or API to assign a floating IP per during installation. This should be avaide. Specificallly the "--floating-ip-enabled" command like option (CLI) and "floating_ip_enabled" parameter (API) should not be configured.

Remove floating IPs from nodes on existing clusters

For existing clusters impacted by this behavior, the simplest workaround is to disassociate Floating IPs from all Kubernetes nodes: - This resolves the conflict by eliminating the competing NAT mechanism. - Can be done via the Rumble Cloud web interface or CLI.

Note: This also disables: - Direct SSH access to the nodes via public IP - NodePort access for Kubernetes services

While this improves security by reducing public exposure, it may break workflows that depend on those access methods.

Avoid re-associating floating IPs

While temporarily re-associating and then disassociating Floating IPs might seem to fix the issue, this is not a long-term solution. The conflict may return, often unpredictably.

Exclude specific nodes from load balancers

If you must use Floating IPs on certain nodes, ensure those nodes are excluded from participating in load balancers. This can be achieved by applying the node.kubernetes.io/exclude-from-external-load-balancers=true label to the approriate nodes. See this article from the Kubernetes documentation for more information on this setting.