Skupper Network Architecture Discoveries

Edge Mode Limitation: Single Active Connection

Discovery

When the local edge site had two outbound links (rhtevan-work and rhel-ai), only one worked. The second link showed octets=0.

Root Cause (Source Code)

In skupper-router/src/router_core/connection_manager.c:

if (cm->active_edge_connection == 0) {
    cm->active_edge_connection = conn;
}

The field active_edge_connection is singular. An edge router maintains only ONE active edge connection at a time. Additional connections open at TCP/TLS level but are never promoted to active.

Documentation Gap

The Skupper docs say “edge sites cannot accept links from remote sites” but do NOT explicitly state they can only have one outbound link. This limitation is only discoverable in source code.

Solution

Changed the local site from edge to interior mode. Interior sites use inter-router connections (not edge connections) and support multiple simultaneous links.

AWS Port Constraints

Problem

The rhel-ai host (bastion.g7cpg.sandbox600.opentlc.com) is behind AWS security groups. Only port 8000 and port 22 are externally reachable. Default Skupper ports (45671, 55671) are blocked.

Discovery Process

  1. Initially assumed port 8000 was only for the model API
  2. Tested connectivity: port 22 ✅, port 8000 ✅, port 45671 ❌
  3. No access to modify AWS security groups

Solution

Remapped the Skupper inter-router listener on rhel-ai from default 55671 to 8000 via custom RouterAccess:

roles:
- name: inter-router
  port: 8000      # only externally reachable port
- name: edge
  port: 45671     # kept for local use

Model API moved from port 8000 to 9000.

Interior Outbound-Only Site Design

The local site needs to connect to two hubs but doesn’t need to accept any inbound connections.

Configuration

apiVersion: skupper.io/v2alpha1
kind: Site
metadata:
  name: local
spec: {}   # no edge: true, no linkAccess → interior, outbound only

This creates an interior router that:

Routing Key & Port Consistency Design

Design Principle

Model serving port = Skupper listener port = routing key suffix identifies the host.

Host Routing Key Model Port Local Listener
rhtevan-work model-api-rhtevan-work 10000 10000
rhel-ai model-api-rhel-ai 9000 9000

Port 8000 is kept unoccupied on localhost — available for other uses and avoids confusion with rhel-ai’s Skupper inter-router port.

Why Two Routing Keys

In Skupper V2, each Listener binds to a specific routingKey AND a specific host:port. You cannot have two Listeners with different routing keys on the same port. One Listener = one routingKey = one port.

RouterAccess Subject Alternative Names (SANs)

When the hub’s TLS certificate is generated, the SANs must include every hostname and IP that clients will use to connect:

subjectAlternativeNames:
- bastion.g7cpg.sandbox600.opentlc.com   # public DNS
- 3.23.208.217                            # public IP
- bastion.g7cpg.internal                  # internal DNS
- 192.168.0.129                           # internal IP
- 127.0.0.1                               # loopback
- 0.0.0.0
- "::"

Missing a SAN causes SSL Failure: certificate verify failed on the connecting site.

When applying link tokens generated by skupper link generate, four manual fixes are needed:

  1. Rename link-hub → link-<site-name> (avoid collisions when multiple links exist)
  2. Fix host — replace 0.0.0.0 with actual hostname/IP
  3. Fix endpoint role — keep only inter-router endpoint, remove edge endpoint for interior-to-interior links
  4. Fix tlsCredentials — must match the renamed Secret name

Skupper CLI Status Reporting Bug

skupper link status, skupper connector status, and skupper listener status consistently report “Pending / Not Operational” and “Not Matched” even when:

This affects both linux and podman platforms (Skupper CLI v2.2.1).

Reliable verification methods:

ss -tnp | grep ESTAB        # link connected?
ss -tlnp | grep 9000        # listener open?
curl localhost:9000/v1/models  # traffic flowing?