PCIeタイプのFPGAを使おうと思うとどうしてもシステムがおおきくなりがちです．ちょっと持ち運んで使いたい場合を想定してGPU拡張ボックスに接続してみました．
フルレングス・フルハイトのGPUも入るボックスなだけあって，Alveo U50は余裕で入ります．なんか，上も結構あいてるから少し前のFPGA評価ボードを入れてもよいかも．
Alveo U50の場合，PCIe x16でデスクトップPCに接続した場合と比べて，転送性能が1/8程度しか出ないものの，とりあえず認識して使えることが確認できました．
環境
試した環境は次の通り
- Lenovo ThinkPad X280（Thunderbolt3対応のUSB-Cポートがある）
- AKiTiO Node Titan (Thunderbolt 3 接続のGPU拡張ボックス)
- OS: Ubuntu 16.04
- FPGAカード: Alveo U50
セットアップ
UEFI(BIOS)でThunderboltのセキュリティチェックをオフにしました．オフにしないと，ThunderboltチップがLinuxで認識しているのにデバイスを認識しないという悲しい状況になります．
なお，boltctlとかインストールするといいんじゃない？というのをイーツリーズの船田さんに教えてもらいましたが，Ubuntu 16.04にさくっとインストールできそうになかったので試してません．
ちなみに，Thunderbolt 3は，コネクタこそUSB Type-Cですが，対応してるポートと対応してないポートがありますので，注意が必要です．X280の場合，拡張ポートを接続する方のUSB Type-Cコネクタの方だけがThunderbolt 3に対応しています．
性能チェック
気になるのは転送性能ですね．デスクトップPCでPCIe x16接続した場合と validate を使って性能を比較してみました．GPUボックスを使うと読み書きが1.5GBps程度でした．PCIe接続では11.9GBps程度なので結構な速度低下です．
GPU拡張ボックスを使った場合
miyo@x280:~$ sudo /opt/xilinx/xrt/bin/xbutil validate
INFO: Found 1 cards
INFO: Validating card[0]: xilinx_u50_gen3x16_xdma_201920_3
INFO: == Starting AUX power connector check:
AUX power connector not available. Skipping validation
INFO: == AUX power connector check SKIPPED
INFO: == Starting PCIE link check:
LINK ACTIVE, ATTENTION
Ensure Card is plugged in to Gen3x16, instead of Gen3x4
Lower performance may be experienced
WARN: == PCIE link check PASSED with warning
INFO: == Starting SC firmware version check:
SC FIRMWARE MISMATCH, ATTENTION
SC firmware running on board: 5.0. Expected SC firmware from installed Shell: 5.0.27
Please use "xbmgmt flash --scan" to check installed Shell.
WARN: == SC firmware version check PASSED with warning
INFO: == Starting verify kernel test:
INFO: == verify kernel test PASSED
INFO: == Starting DMA test:
Host -> PCIe -> FPGA write bandwidth = 1568.04 MB/s
Host <- PCIe <- FPGA read bandwidth = 1416 MB/s
INFO: == DMA test PASSED
INFO: == Starting device memory bandwidth test:
...........
Maximum throughput: 52428 MB/s
INFO: == device memory bandwidth test PASSED
INFO: == Starting PCIE peer-to-peer test:
P2P BAR is not enabled. Skipping validation
INFO: == PCIE peer-to-peer test SKIPPED
INFO: == Starting memory-to-memory DMA test:
M2M is not available. Skipping validation
INFO: == memory-to-memory DMA test SKIPPED
INFO: Card[0] validated with warnings.
INFO: All cards validated successfully but with warnings.
PCIeに直接接続した場合
miyo@dev-8800:~$ sudo /opt/xilinx/xrt/bin/xbutil validate
[sudo] password for miyo:
INFO: Found 1 cards
INFO: Validating card[0]: xilinx_u50_gen3x16_xdma_201920_3
INFO: == Starting AUX power connector check:
AUX power connector not available. Skipping validation
INFO: == AUX power connector check SKIPPED
INFO: == Starting PCIE link check:
INFO: == PCIE link check PASSED
INFO: == Starting SC firmware version check:
SC FIRMWARE MISMATCH, ATTENTION
SC firmware running on board: 5.0. Expected SC firmware from installed Shell: 5.0.27
Please use "xbmgmt flash --scan" to check installed Shell.
WARN: == SC firmware version check PASSED with warning
INFO: == Starting verify kernel test:
INFO: == verify kernel test PASSED
INFO: == Starting DMA test:
Host -> PCIe -> FPGA write bandwidth = 11855.7 MB/s
Host <- PCIe <- FPGA read bandwidth = 11839.2 MB/s
INFO: == DMA test PASSED
INFO: == Starting device memory bandwidth test:
...........
Maximum throughput: 52428 MB/s
INFO: == device memory bandwidth test PASSED
INFO: == Starting PCIE peer-to-peer test:
P2P BAR is not enabled. Skipping validation
INFO: == PCIE peer-to-peer test SKIPPED
INFO: == Starting memory-to-memory DMA test:
M2M is not available. Skipping validation
INFO: == memory-to-memory DMA test SKIPPED
INFO: Card[0] validated with warnings.
INFO: All cards validated successfully but with warnings.
miyo@dev-8800:~$
まとめ
転送性能は，当たり前のように大きく低下します．ただ，ポータブルで利用できるっていうのは便利ですね．特に”100GbEを使って何かと接続する”，というような用途にはフィットしそうです．
なお，GPUボックスはGPU自体にファンがついていることが想定されているようです．ケースにファンはついていますが，Alveo U50はまったく冷却できませんので注意が必要です．ちょっと評価するだけだけだから，と油断してしまったら，すごくすごく熱くなりました…
おまけ; 認識してる様子
こんな感じで認識してました．
02:00.0 PCI bridge: Intel Corporation JHL6240 Thunderbolt 3 Bridge (Low Power) [Alpine Ridge LP 2016] (rev 01) (prog-if 00 [Normal decode])
Physical Slot: 0
Flags: bus master, fast devsel, latency 0, IRQ 16
Bus: primary=02, secondary=03, subordinate=3a, sec-latency=0
I/O behind bridge: 00002000-00002fff
Memory behind bridge: d0000000-e60fffff
Prefetchable memory behind bridge: 0000000090000000-00000000b1ffffff
Capabilities: [80] Power Management version 3
Capabilities: [88] MSI: Enable- Count=1/1 Maskable- 64bit+
Capabilities: [ac] Subsystem: Device 2222:1111
Capabilities: [c0] Express Upstream Port, MSI 00
Capabilities: [100] Device Serial Number 26-bc-c0-62-88-b7-d0-00
Capabilities: [200] Advanced Error Reporting
Capabilities: [300] Virtual Channel
Capabilities: [400] Power Budgeting <?>
Capabilities: [500] Vendor Specific Information: ID=1234 Rev=1 Len=0e0 <?>
Capabilities: [600] Latency Tolerance Reporting
Capabilities: [700] #19
Kernel driver in use: pcieport
Kernel modules: shpchp
03:00.0 PCI bridge: Intel Corporation JHL6240 Thunderbolt 3 Bridge (Low Power) [Alpine Ridge LP 2016] (rev 01) (prog-if 00 [Normal decode])
Flags: bus master, fast devsel, latency 0, IRQ 16
Bus: primary=03, secondary=04, subordinate=04, sec-latency=0
Memory behind bridge: e6000000-e60fffff
Capabilities: [80] Power Management version 3
Capabilities: [88] MSI: Enable- Count=1/1 Maskable- 64bit+
Capabilities: [ac] Subsystem: Device 2222:1111
Capabilities: [c0] Express Downstream Port (Slot+), MSI 00
Capabilities: [100] Device Serial Number 26-bc-c0-62-88-b7-d0-00
Capabilities: [200] Advanced Error Reporting
Capabilities: [300] Virtual Channel
Capabilities: [400] Power Budgeting <?>
Capabilities: [500] Vendor Specific Information: ID=1234 Rev=1 Len=0e0 <?>
Capabilities: [700] #19
Kernel driver in use: pcieport
Kernel modules: shpchp
03:01.0 PCI bridge: Intel Corporation JHL6240 Thunderbolt 3 Bridge (Low Power) [Alpine Ridge LP 2016] (rev 01) (prog-if 00 [Normal decode])
Flags: bus master, fast devsel, latency 0, IRQ 125
Bus: primary=03, secondary=05, subordinate=39, sec-latency=0
I/O behind bridge: 00002000-00002fff
Memory behind bridge: d0000000-e5efffff
Prefetchable memory behind bridge: 0000000090000000-00000000b1ffffff
Capabilities: [80] Power Management version 3
Capabilities: [88] MSI: Enable+ Count=1/1 Maskable- 64bit+
Capabilities: [ac] Subsystem: Device 2222:1111
Capabilities: [c0] Express Downstream Port (Slot+), MSI 00
Capabilities: [100] Device Serial Number 26-bc-c0-62-88-b7-d0-00
Capabilities: [200] Advanced Error Reporting
Capabilities: [300] Virtual Channel
Capabilities: [400] Power Budgeting <?>
Capabilities: [500] Vendor Specific Information: ID=1234 Rev=1 Len=0e0 <?>
Capabilities: [700] #19
Kernel driver in use: pcieport
Kernel modules: shpchp
03:02.0 PCI bridge: Intel Corporation JHL6240 Thunderbolt 3 Bridge (Low Power) [Alpine Ridge LP 2016] (rev 01) (prog-if 00 [Normal decode])
Flags: bus master, fast devsel, latency 0, IRQ 18
Bus: primary=03, secondary=3a, subordinate=3a, sec-latency=0
Memory behind bridge: e5f00000-e5ffffff
Capabilities: [80] Power Management version 3
Capabilities: [88] MSI: Enable- Count=1/1 Maskable- 64bit+
Capabilities: [ac] Subsystem: Device 2222:1111
Capabilities: [c0] Express Downstream Port (Slot+), MSI 00
Capabilities: [100] Device Serial Number 26-bc-c0-62-88-b7-d0-00
Capabilities: [200] Advanced Error Reporting
Capabilities: [300] Virtual Channel
Capabilities: [400] Power Budgeting <?>
Capabilities: [500] Vendor Specific Information: ID=1234 Rev=1 Len=0e0 <?>
Capabilities: [700] #19
Kernel driver in use: pcieport
Kernel modules: shpchp
04:00.0 System peripheral: Intel Corporation JHL6240 Thunderbolt 3 NHI (Low Power) [Alpine Ridge LP 2016] (rev 01)
Subsystem: Device 2222:1111
Flags: bus master, fast devsel, latency 0, IRQ 16
Memory at e6000000 (32-bit, non-prefetchable) [size=256K]
Memory at e6040000 (32-bit, non-prefetchable) [size=4K]
Capabilities: [80] Power Management version 3
Capabilities: [88] MSI: Enable- Count=1/1 Maskable- 64bit+
Capabilities: [c0] Express Endpoint, MSI 00
Capabilities: [a0] MSI-X: Enable+ Count=16 Masked-
Capabilities: [100] Device Serial Number 26-bc-c0-62-88-b7-d0-00
Capabilities: [200] Advanced Error Reporting
Capabilities: [300] Virtual Channel
Capabilities: [400] Power Budgeting <?>
Capabilities: [500] Vendor Specific Information: ID=1234 Rev=1 Len=088 <?>
Capabilities: [600] Latency Tolerance Reporting
Kernel driver in use: thunderbolt
Kernel modules: thunderbolt
05:00.0 PCI bridge: Intel Corporation JHL7540 Thunderbolt 3 Bridge [Titan Ridge DD 2018] (rev 06) (prog-if 00 [Normal decode])
Physical Slot: 1
Flags: bus master, fast devsel, latency 0, IRQ 17
Bus: primary=05, secondary=06, subordinate=39, sec-latency=0
I/O behind bridge: 00002000-00002fff
Memory behind bridge: d0000000-e5efffff
Prefetchable memory behind bridge: 0000000090000000-00000000b1ffffff
Capabilities: [80] Power Management version 3
Capabilities: [88] MSI: Enable- Count=1/1 Maskable- 64bit+
Capabilities: [ac] Subsystem: Device 1cf0:de31
Capabilities: [c0] Express Upstream Port, MSI 00
Capabilities: [50] #15 [0000]
Capabilities: [100] Device Serial Number 10-9f-a4-32-6b-b7-d0-00
Capabilities: [200] Advanced Error Reporting
Capabilities: [300] Virtual Channel
Capabilities: [400] Power Budgeting <?>
Capabilities: [500] Vendor Specific Information: ID=1234 Rev=1 Len=100 <?>
Capabilities: [600] Vendor Specific Information: ID=8086 Rev=2 Len=04c <?>
Capabilities: [700] #19
Capabilities: [800] Latency Tolerance Reporting
Capabilities: [a00] L1 PM Substates
Capabilities: [b00] #1f
Kernel driver in use: pcieport
Kernel modules: shpchp
06:01.0 PCI bridge: Intel Corporation JHL7540 Thunderbolt 3 Bridge [Titan Ridge DD 2018] (rev 06) (prog-if 00 [Normal decode])
Flags: bus master, fast devsel, latency 0, IRQ 18
Bus: primary=06, secondary=07, subordinate=07, sec-latency=0
Prefetchable memory behind bridge: 0000000090000000-00000000afffffff
Capabilities: [80] Power Management version 3
Capabilities: [88] MSI: Enable- Count=1/1 Maskable- 64bit+
Capabilities: [ac] Subsystem: Device 1cf0:de31
Capabilities: [c0] Express Downstream Port (Slot+), MSI 00
Capabilities: [50] #15 [0000]
Capabilities: [100] Device Serial Number 10-9f-a4-32-6b-b7-d0-00
Capabilities: [200] Advanced Error Reporting
Capabilities: [300] Virtual Channel
Capabilities: [400] Power Budgeting <?>
Capabilities: [500] Vendor Specific Information: ID=1234 Rev=1 Len=100 <?>
Capabilities: [600] Vendor Specific Information: ID=8086 Rev=2 Len=04c <?>
Capabilities: [700] #19
Capabilities: [900] Access Control Services
Kernel driver in use: pcieport
Kernel modules: shpchp
06:02.0 PCI bridge: Intel Corporation JHL7540 Thunderbolt 3 Bridge [Titan Ridge DD 2018] (rev 06) (prog-if 00 [Normal decode])
Flags: bus master, fast devsel, latency 0, IRQ 19
Bus: primary=06, secondary=08, subordinate=08, sec-latency=0
Memory behind bridge: d0000000-d00fffff
Capabilities: [80] Power Management version 3
Capabilities: [88] MSI: Enable- Count=1/1 Maskable- 64bit+
Capabilities: [ac] Subsystem: Device 1cf0:de31
Capabilities: [c0] Express Downstream Port (Slot+), MSI 00
Capabilities: [50] #15 [0000]
Capabilities: [100] Device Serial Number 10-9f-a4-32-6b-b7-d0-00
Capabilities: [200] Advanced Error Reporting
Capabilities: [300] Virtual Channel
Capabilities: [400] Power Budgeting <?>
Capabilities: [500] Vendor Specific Information: ID=1234 Rev=1 Len=100 <?>
Capabilities: [600] Vendor Specific Information: ID=8086 Rev=2 Len=04c <?>
Capabilities: [700] #19
Capabilities: [900] Access Control Services
Kernel driver in use: pcieport
Kernel modules: shpchp
06:04.0 PCI bridge: Intel Corporation JHL7540 Thunderbolt 3 Bridge [Titan Ridge DD 2018] (rev 06) (prog-if 00 [Normal decode])
Flags: bus master, fast devsel, latency 0, IRQ 126
Bus: primary=06, secondary=09, subordinate=39, sec-latency=0
I/O behind bridge: 00002000-00002fff
Memory behind bridge: d0100000-e5efffff
Prefetchable memory behind bridge: 00000000b0000000-00000000b1ffffff
Capabilities: [80] Power Management version 3
Capabilities: [88] MSI: Enable+ Count=1/1 Maskable- 64bit+
Capabilities: [ac] Subsystem: Device 1cf0:de31
Capabilities: [c0] Express Downstream Port (Slot+), MSI 00
Capabilities: [50] #15 [0000]
Capabilities: [100] Device Serial Number 10-9f-a4-32-6b-b7-d0-00
Capabilities: [200] Advanced Error Reporting
Capabilities: [300] Virtual Channel
Capabilities: [400] Power Budgeting <?>
Capabilities: [500] Vendor Specific Information: ID=1234 Rev=1 Len=100 <?>
Capabilities: [600] Vendor Specific Information: ID=8086 Rev=2 Len=04c <?>
Capabilities: [700] #19
Capabilities: [900] Access Control Services
Kernel driver in use: pcieport
Kernel modules: shpchp
07:00.0 Processing accelerators: Xilinx Corporation Device 5020
Subsystem: Xilinx Corporation Device 000e
Flags: bus master, fast devsel, latency 0
Memory at 90000000 (64-bit, prefetchable) [size=32M]
Memory at 92000000 (64-bit, prefetchable) [size=128K]
Capabilities: [40] Power Management version 3
Capabilities: [60] MSI-X: Enable- Count=33 Masked-
Capabilities: [70] Express Endpoint, MSI 00
Capabilities: [100] Advanced Error Reporting
Capabilities: [1c0] #19
Capabilities: [e00] Access Control Services
Kernel driver in use: xclmgmt
Kernel modules: xclmgmt
07:00.1 Processing accelerators: Xilinx Corporation Device 5021
Subsystem: Xilinx Corporation Device 000e
Flags: bus master, fast devsel, latency 0, IRQ 18
Memory at 94000000 (64-bit, prefetchable) [size=32M]
Memory at 96000000 (64-bit, prefetchable) [size=64K]
Memory at a0000000 (64-bit, prefetchable) [size=256M]
Capabilities: [40] Power Management version 3
Capabilities: [60] MSI-X: Enable- Count=33 Masked-
Capabilities: [70] Express Endpoint, MSI 00
Capabilities: [100] Advanced Error Reporting
Capabilities: [e00] Access Control Services
Kernel driver in use: xocl
Kernel modules: xocl
08:00.0 USB controller: Intel Corporation JHL7540 Thunderbolt 3 USB Controller [Titan Ridge DD 2018] (rev 06) (prog-if 30 [XHCI])
Subsystem: Device 1cf0:de31
Flags: bus master, fast devsel, latency 0, IRQ 128
Memory at d0000000 (32-bit, non-prefetchable) [size=64K]
Capabilities: [80] Power Management version 3
Capabilities: [88] MSI: Enable+ Count=1/8 Maskable- 64bit+
Capabilities: [c0] Express Endpoint, MSI 00
Capabilities: [100] Device Serial Number 10-9f-a4-32-6b-b7-d0-00
Capabilities: [200] Advanced Error Reporting
Capabilities: [300] Virtual Channel
Capabilities: [400] Power Budgeting <?>
Capabilities: [500] Vendor Specific Information: ID=1234 Rev=1 Len=100 <?>
Capabilities: [600] Vendor Specific Information: ID=8086 Rev=2 Len=04c <?>
Capabilities: [700] #19
Capabilities: [800] Latency Tolerance Reporting
Kernel driver in use: xhci_hcd