ASUS Ascent GX10 (DGX Spark) 2台をConnect-Xで繋ぐメモ

ASUS Ascent GX10 (DGX Spark) 2台をConnect-Xで繋ぐメモです。

まず、2台をQSFP Link ケーブルで繋ぎます。

RoCEインターフェースとLinuxのネットワークデバイスとのマッピングの確認

1台目

$ ibdev2netdev
rocep1s0f0 port 1 ==> enp1s0f0np0 (Up)
rocep1s0f1 port 1 ==> enp1s0f1np1 (Down)
roceP2p1s0f0 port 1 ==> enP2p1s0f0np0 (Up)
roceP2p1s0f1 port 1 ==> enP2p1s0f1np1 (Down)

2台目

$ ibdev2netdev
rocep1s0f0 port 1 ==> enp1s0f0np0 (Up)
rocep1s0f1 port 1 ==> enp1s0f1np1 (Down)
roceP2p1s0f0 port 1 ==> enP2p1s0f0np0 (Up)
roceP2p1s0f1 port 1 ==> enP2p1s0f1np1 (Down)

Netplan設定ファイルの作成

1台目

sudo tee /etc/netplan/40-cx7.yaml > /dev/null <<EOF
network:
  version: 2
  ethernets:
    enp1s0f0np0:
      addresses:
      - 192.168.100.20/24
      dhcp4: no
    enP2p1s0f0np0:
      addresses:
      - 192.168.101.20/24
      dhcp4: no
EOF
sudo chmod 600 /etc/netplan/40-cx7.yaml
sudo netplan apply

2台目

sudo tee /etc/netplan/40-cx7.yaml > /dev/null <<EOF
network:
  version: 2
  ethernets:
    enp1s0f0np0:
      addresses:
      - 192.168.100.30/24
      dhcp4: no
    enP2p1s0f0np0:
      addresses:
      - 192.168.101.30/24
      dhcp4: no
EOF
sudo chmod 600 /etc/netplan/40-cx7.yaml
sudo netplan apply

デバイス情報を確認

$ nmcli device
DEVICE          TYPE      STATE                   CONNECTION            
enP7s7          ethernet  connected               Wired connection 3    
enP2p1s0f0np0   ethernet  connected               netplan-enP2p1s0f0np0 
enp1s0f0np0     ethernet  connected               netplan-enp1s0f0np0   
lo              loopback  connected (externally)  lo                    
docker0         bridge    connected (externally)  docker0               
wlP9s9          wifi      disconnected            --                    
p2p-dev-wlP9s9  wifi-p2p  disconnected            --                    
enP2p1s0f1np1   ethernet  unavailable             --                    
enp1s0f1np1     ethernet  unavailable             --