Skip to content

Network Diagram

Physical Network Topology

graph TB
    subgraph ToR["Top-of-Rack Switch"]
        USW["Switch<br/>(LACP + VLANs)"]
    end

    subgraph Nodes["MS-01 Bare Metal Nodes"]
        subgraph N1["txulspectropoc1 (10.0.12.14)"]
            NIC1A["enp55s0f0np0"] --> BOND1["bond0 (LACP)"]
            NIC1B["enp93s0f1np1"] --> BOND1
            BOND1 --> BR1["br0 (bridge)"]
            BR1 --> VLAN1["br0.13 (VLAN 13)"]
        end
        subgraph N2["txulspectropoc2 (10.0.12.18)"]
            NIC2A["enp55s0f0np0"] --> BOND2["bond0 (LACP)"]
            NIC2B["enp93s0f1np1"] --> BOND2
            BOND2 --> BR2["br0 (bridge)"]
            BR2 --> VLAN2["br0.13 (VLAN 13)"]
        end
        subgraph N3["txulspectropoc3 (10.0.12.19)"]
            NIC3A["enp55s0f0np0"] --> BOND3["bond0 (LACP)"]
            NIC3B["enp93s0f1np1"] --> BOND3
            BOND3 --> BR3["br0 (bridge)"]
            BR3 --> VLAN3["br0.13 (VLAN 13)"]
        end
    end

    USW --- NIC1A
    USW --- NIC1B
    USW --- NIC2A
    USW --- NIC2B
    USW --- NIC3A
    USW --- NIC3B

    subgraph Net["Network Services"]
        GW["Gateway: 10.0.12.1"]
        DNS["DNS: 10.0.11.2 / .4 / .5"]
        PXA["Pure FlashArray (FC)"]
    end

    VLAN1 --- GW
    VLAN2 --- GW
    VLAN3 --- GW

Network Stack (Per Node)

Physical NICs (enp55s0f0np0 + enp93s0f1np1)
    │
    ├── bond0 (802.3ad LACP)
    │       mode: 802.3ad
    │       lacp-rate: fast
    │       mii-monitor-interval: 100
    │       transmit-hash-policy: layer3+4
    │
    ├── br0 (Linux Bridge, STP disabled)
    │       forward-delay: 0
    │
    └── br0.13 (VLAN 13)
            address: 10.0.12.x/24
            gateway: 10.0.12.1
            dns: 10.0.11.2, 10.0.11.5, 10.0.11.4
            search: mouser.lan

Kubernetes Networking

Pod Network (Cilium)

Property Value
CNI Cilium 1.18.4
Pod CIDR Palette variable: {{ .spectro.var.K8sPodCIDR }}
Pod Mask Size /24 per node
IPAM Mode Cluster Pool

VM Network (Multus + Bridge)

VMs connect directly to the physical network via Multus with a bridge CNI plugin. This gives VMs their own IPs on the VLAN 13 subnet (10.0.12.x).

graph LR
    subgraph VM["Virtual Machine"]
        ETH["eth0 (virtio)"]
    end
    subgraph Pod["virt-launcher Pod"]
        TAP["tap device"]
        BRIDGE_POD["br0 (pod bridge)"]
    end
    subgraph Node["Host Node"]
        BR0["br0 (host bridge)"]
        BOND["bond0 (LACP)"]
    end

    ETH --> TAP --> BRIDGE_POD --> BR0 --> BOND

NetworkAttachmentDefinitions

NAD Namespace Type Bridge VLAN
vlan-13 default bridge br0 13
vlan-12 default bridge br0 12

Cross-Namespace Access

NADs in the default namespace are accessible from any namespace using the format default/vlan-13 in the VM spec.

Load Balancer (MetalLB)

Property Value
Mode L2
Auto-Assign false (IPs must be explicitly requested)
Ingress IP 10.0.12.36 (nginx LoadBalancer)

autoAssign is disabled

MetalLB autoAssign is set to false in this environment. Services of type LoadBalancer must specify spec.loadBalancerIP or use an annotation to request a specific IP from the pool.

Key Service IPs

Service IP Namespace
nginx Ingress 10.0.12.36 ingress-nginx

Netplan Reference (Per Node)

network:
  version: 2
  renderer: networkd

  ethernets:
    enp55s0f0np0:
      optional: true
      dhcp4: false
    enp93s0f1np1:
      optional: true
      dhcp4: false

  bonds:
    bond0:
      interfaces:
        - enp55s0f0np0
        - enp93s0f1np1
      parameters:
        mode: 802.3ad
        lacp-rate: fast
        mii-monitor-interval: 100
        transmit-hash-policy: layer3+4

  bridges:
    br0:
      interfaces:
        - bond0
      parameters:
        stp: false
        forward-delay: 0

  vlans:
    br0.13:
      id: 13
      link: br0
      addresses:
        - 10.0.12.x/24        # .14, .18, or .19 depending on node
      routes:
        - to: default
          via: 10.0.12.1
      nameservers:
        search: [mouser.lan]
        addresses:
          - 10.0.11.2
          - 10.0.11.5
          - 10.0.11.4