Alveo U50のセットアップをする

このエントリーをはてなブックマークに追加
Pocket

HBMを搭載したAlveo U50をセットアップしてみました．セットアップ対象は，x86_64の普通のパソコンです．スペックは，

  • CPU: Intel(R) Core(TM) i5-8400 CPU @ 2.80GHz
  • メモリ: MemTotal: 64GB  DDR4 2400 MT/s

OSはUbuntuで18.04で，詳細は次の通り．

$ uname -a
Linux dev-8800 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
$ cat /etc/lsb-release 
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=18.04
DISTRIB_CODENAME=bionic
DISTRIB_DESCRIPTION="Ubuntu 18.04.3 LTS"

Vitisの正式サポートはUbuntu 18.04.2までなのですが，とりあえず動作は確認できました．
重要なのは，libjson-glibとunzipのインストールです．

インストールの準備

libjson-glib-1.0-0とunzipがないと，あとで，xbutil validate と v++ を実行する時にコケるので

$ sudo apt install libjson-glib-1.0-0
$ sudo apt install unzip

としてインストールしておきましょう．

また，一通りセットアップして動作確認した後で，Ubuntuの自動アップデートで動かなくなると嫌なので，

$ cat /etc/apt/apt.conf.d/20auto-upgrades 
APT::Periodic::Update-Package-Lists "0";
APT::Periodic::Unattended-Upgrade "0";

と，自動アップデートはオフにしておきます．

Alveo U50をパソコンに接続・セットアップ

Alveo U50 Data Center Accelerator Card Installation Guide に従ってインストールします．ランタイムやプラットフォーム定義ファイルは /opt/xilinx にインストールされます．ちなみに，他の場所へのシンボリックリンクにしておくと，うまくスクリプトが動作しないことがありました．おとなしくそのままにするのがよいでしょう．

まずは，Alveo U50のGetting StartedからRuntimeとDeployment Target Platform，Development Target Platformの

  • xrt_201920.2.3.1301_18.04-xrt.deb
  • xilinx-u50-xdma-201920.1-2699728_18.04.deb
  • xilinx-u50-xdma-dev-201920.1-2699728_18.04.deb

をダウンロードしてインストール．

$ sudo apt install ./xrt_201920.2.3.1301_18.04-xrt.deb
$ sudo apt install ./xilinx-u50-xdma-201920.1-2699728_18.04.deb
$ sudo apt install ./xilinx-u50-xdma-dev-201920.1-2699728_18.04.deb

aptでdebパッケージインストールするときは ./ が必要なので注意．
xilinx-u50-xdma-201920.1-2699728_18.04.debをインストールすると表示されるメッセージに従って，

$ sudo /opt/xilinx/xrt/bin/xbmgmt flash --update --shell xilinx_u50_xdma_201920_1

を実行．実行中はCtrl-C押すなって書いてあるので，間違っても押さないように注意．
こんな感じで処理がすすむ．

$ sudo /opt/xilinx/xrt/bin/xbmgmt flash --update --shell xilinx_u50_xdma_201920_1
Status: shell needs updating
Current shell: xilinx_u50_GOLDEN_9
Shell to be flashed: xilinx_u50_xdma_201920_1
Are you sure you wish to proceed? [y/n]: y

Updating SC firmware on card[0000:01:00.0]
INFO: found 5 sections
..............................
INFO: Loading new firmware on SC

Updating shell on card[0000:01:00.0]
INFO: ***Found 507 ELA Records
Idcode byte[0] ff
Idcode byte[1] 20
Idcode byte[2] bb
Idcode byte[3] 21
Idcode byte[4] 10
Enabled bitstream guard. Bitstream will not be loaded until flashing is finished.
Erasing flash.........................
Programming flash.........................
Cleared bitstream guard. Bitstream now active.
Successfully flashed Card[0000:01:00.0]

1 Card(s) flashed successfully.
Cold reboot machine to load the new image on card(s).

電源落として再起動．lspciで確認すると，

01:00.0 Processing accelerators: Xilinx Corporation Device 5020
Subsystem: Xilinx Corporation Device 000e
Flags: bus master, fast devsel, latency 0
Memory at b2000000 (64-bit, prefetchable) [size=32M]
Memory at b4000000 (64-bit, prefetchable) [size=128K]
Capabilities: [40] Power Management version 3
Capabilities: [60] MSI-X: Enable+ Count=33 Masked-
Capabilities: [70] Express Endpoint, MSI 00
Capabilities: [100] Advanced Error Reporting
Capabilities: [1c0] #19
Capabilities: [e00] Access Control Services
Kernel driver in use: xclmgmt
Kernel modules: xclmgmt

01:00.1 Processing accelerators: Xilinx Corporation Device 5021
Subsystem: Xilinx Corporation Device 000e
Flags: bus master, fast devsel, latency 0, IRQ 16
Memory at b0000000 (64-bit, prefetchable) [size=32M]
Memory at b4020000 (64-bit, prefetchable) [size=64K]
Memory at a0000000 (64-bit, prefetchable) [size=256M]
Capabilities: [40] Power Management version 3
Capabilities: [60] MSI-X: Enable+ Count=33 Masked-
Capabilities: [70] Express Endpoint, MSI 00
Capabilities: [100] Advanced Error Reporting
Capabilities: [e00] Access Control Services
Kernel driver in use: xocl
Kernel modules: xocl

と，デバイスがみえた．

Alveo U50のファームウェアの情報を確認

xbmgmtでフラッシュの情報を確認することができます．

$ sudo /opt/xilinx/xrt/bin/xbmgmt flash --scan
Card [0000:01:00.0]
Card type: u50
Flash type: SPI
Flashable partition running on FPGA:
xilinx_u50_xdma_201920_1,[ID=0x000000005daa78ef],[SC=5.0.13]
Flashable partitions installed in system:
xilinx_u50_xdma_201920_1,[ID=0x000000005daa78ef],[SC=5.0.13]

Alveo U50の動作確認

validateコマンドで動作確認，簡単な性能の確認ができます．

$ sudo /opt/xilinx/xrt/bin/xbutil validate
INFO: Found 1 cards

INFO: Validating card[0]: xilinx_u50_xdma_201920_1
INFO: == Starting AUX power connector check:
AUX power connector not available. Skipping validation
INFO: == AUX power connector check SKIPPED
INFO: == Starting PCIE link check:
INFO: == PCIE link check PASSED
INFO: == Starting verify kernel test:
INFO: == verify kernel test PASSED
INFO: == Starting DMA test:

Host -> PCIe -> FPGA (HBM[0]) write BW: Average: 12064 MB/s
Host <- PCIe <- FPGA (HBM[0]) read BW: Average: 11745 MB/s
INFO: == DMA test PASSED
INFO: == Starting device memory bandwidth test:
.....
FPGA <- HBM (8 channel(s) sum) Average Read Bandwidth: 50415.092001 MBps FPGA -> HBM (8 channel(s) sum) Average Write Bandwidth: 50581.614236 MBps
INFO: == device memory bandwidth test PASSED
INFO: Card[0] validated successfully.

INFO: All cards validated successfully.

もし，libjson-glib-1.0をインストールしていない場合には，

$ sudo /opt/xilinx/xrt/bin/xbutil validate
INFO: Found 1 cards

INFO: Validating card[0]: xilinx_u50_xdma_201920_1
INFO: == Starting AUX power connector check:
AUX power connector not available. Skipping validation
INFO: == AUX power connector check SKIPPED
INFO: == Starting PCIE link check:
INFO: == PCIE link check PASSED
INFO: == Starting verify kernel test:
/opt/xilinx/dsa/xilinx_u50_xdma_201920_1/test/xbtest: error while loading shared libraries: libjson-glib-1.0.so.0: cannot open shared object file: No such file or directory

ERROR: == verify kernel test FAILED
INFO: Card[0] failed to validate.

ERROR: Some cards failed to validate.

と怒られます．

$ sudo apt install libjson-glib-1.0-0

とかして，再度実行しましょう．

Vitisのインストール

Vitisのインストールに先立って，OpenCL関連のライブラリをインストールしておきます．

$ sudo apt install ocl-icd-libopencl1 opencl-headers ocl-icd-opencl-dev

あとは，VitisをGUIでインストール（ここでは， /home/tools/Xilinx/ の 下にrootでインストールした）．インストールしたら，ドキュメントに注意書きがあるようにパーミッション関係のトラブルを回避するべく，

$ chmod -R o=g /Vitis/2019.2/tps/lnx64/jre9.0.4
$ chmod -R o=g /Vivado/2019.2/tps/lnx64/jre9.0.4
$ chmod -R o=g /.xinstall/Vitis_2019.2/tps/lnx64/jre9.0.4

としておきます．

SWエミュレーションで，CXXABI_1.3.11がない，とかいっておこられることがある．これはVitisが抱えているlibstdc++のバージョンの問題．システムのlibstdc++に置換するとよい．

$ cd /home/tools/Xilinx/Vitis/2019.2/lib/lnx64.o/Default
$ sudo mv libstdc++.so.6 libstdc++.so.6.old
$ sudo cp /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.25 .
$ sudo ln -sf libstdc++.so.6.0.25 libstdc++.so.6

合成環境の確認

Fixstars TechBlogの 2019.2 Vitis(TM) チュートリアルやってみた に取りあげられてる
https://github.com/Xilinx/Vitis-Tutorials/ の Pathway3で，合成環境の確認をしてみることにします．

$ git clone https://github.com/Xilinx/Vitis-Tutorials.git
$ cd Vitis-Tutorials/docs/Pathway3/reference-files/run

とチュートリアルをcloneして作業ディレクトリに移動．U50で動かしたいので，次のような内容にdesign.cfgを書き変えます．

$ cat design.cfg
platform=xilinx_u50_xdma_201920_1
debug=1

[connectivity]
nk=mmult:1:mmult_1

で，細かいことはおいておいて合成．

$ make build TARGET=hw PLATFORM=xilinx_u50_xdma_201920_1

無事に合成がおわると，hostとmmult.hw.xilinx_u50_xdma_201920_1.xclbinが作成される．

もし，↓のようなエラーがでた場合はunzipが入ってないので，apt install unzipとかする．

...
INFO: [v++ 60-1306] Additional information associated with this v++ link can be found at:
Reports: /home/miyo/work/Vitis-Tutorials/docs/Pathway3/reference-files/run/_x/reports/link
Log files: /home/miyo/work/Vitis-Tutorials/docs/Pathway3/reference-files/run/_x/logs/link
INFO: [v++ 60-1548] Creating build summary session with primary output /home/miyo/work/Vitis-Tutorials/docs/Pathway3/reference-files/run/mmult.hw.xilinx_u50_xdma_201920_1.xclbin.link_summary, at Sun Jan 19 07:16:03 2020
INFO: [v++ 60-1316] Initiating connection to rulecheck server, at Sun Jan 19 07:16:03 2020
INFO: [v++ 60-1315] Creating rulecheck session with output '/home/miyo/work/Vitis-Tutorials/docs/Pathway3/reference-files/run/_x/reports/link/v++_link_mmult.hw.xilinx_u50_xdma_201920_1_guidance.html', at Sun Jan 19 07:16:04 2020
INFO: [v++ 60-895] Target platform: /opt/xilinx/platforms/xilinx_u50_xdma_201920_1/xilinx_u50_xdma_201920_1.xpfm
INFO: [v++ 60-1578] This platform contains Device Support Archive '/opt/xilinx/platforms/xilinx_u50_xdma_201920_1/hw/xilinx_u50_xdma_201920_1.dsa'
INFO: [v++ 60-629] Linking for hardware target
INFO: [v++ 60-423] Target device: xilinx_u50_xdma_201920_1
INFO: [v++ 60-1332] Run 'run_link' status: Not started
INFO: [v++ 60-1443] [07:16:09] Run run_link: Step system_link: Started
INFO: [v++ 60-1453] Command Line: system_link --xo /home/miyo/work/Vitis-Tutorials/docs/Pathway3/reference-files/run/mmult.hw.xilinx_u50_xdma_201920_1.xo --config /home/miyo/work/Vitis-Tutorials/docs/Pathway3/reference-files/run/_x/link/int/syslinkConfig.ini --xpfm /opt/xilinx/platforms/xilinx_u50_xdma_201920_1/xilinx_u50_xdma_201920_1.xpfm --target hw --output_dir /home/miyo/work/Vitis-Tutorials/docs/Pathway3/reference-files/run/_x/link/int --temp_dir /home/miyo/work/Vitis-Tutorials/docs/Pathway3/reference-files/run/_x/link/sys_link
INFO: [v++ 60-1454] Run Directory: /home/miyo/work/Vitis-Tutorials/docs/Pathway3/reference-files/run/_x/link/run_link
ERROR: [SYSTEM_LINK 82-39] Unable to run blocking command: command not found: -o -j /opt/xilinx/platforms/xilinx_u50_xdma_201920_1/hw/xilinx_u50_xdma_201920_1.dsa dsa.xml -d /home/miyo/work/Vitis-Tutorials/docs/Pathway3/reference-files/run/_x/link/sys_link
ERROR: [SYSTEM_LINK 82-87] Unable to read hardware platform file /opt/xilinx/platforms/xilinx_u50_xdma_201920_1/hw/xilinx_u50_xdma_201920_1.dsa and extract meta-data file
INFO: [SYSTEM_LINK 60-1316] Initiating connection to rulecheck server, at Sun Jan 19 07:16:10 2020
ERROR: [SYSTEM_LINK 82-89] Unable to read platform default clock ID to resolve clock ID
ERROR: [SYSTEM_LINK 82-90] Unable to resolve clock ID
ERROR: [SYSTEM_LINK 82-66] Error processing .xo files
ERROR: [SYSTEM_LINK 82-100] Error processing object files, exiting
INFO: [v++ 60-1442] [07:16:10] Run run_link: Step system_link: Failed
Time (s): cpu = 00:00:00.80 ; elapsed = 00:00:00.84 . Memory (MB): peak = 677.559 ; gain = 0.000 ; free physical = 17100 ; free virtual = 70170
ERROR: [v++ 60-661] v++ link run 'run_link' failed
ERROR: [v++ 60-626] Kernel link failed to complete
ERROR: [v++ 60-703] Failed to finish linking

無事に合成がおわったら実行してみましょう．

$ sudo -s
# source /home/tools/Xilinx/Vitis/2019.2/settings64.sh
# source /opt/xilinx/xrt/setup.sh
# ./host mmult.hw.xilinx_u50_xdma_201920_1.xclbin

として実行できます．次のように．

# ./host mmult.hw.xilinx_u50_xdma_201920_1.xclbin
Found Platform
Platform Name: Xilinx
INFO: Reading mmult.hw.xilinx_u50_xdma_201920_1.xclbin
Loading: 'mmult.hw.xilinx_u50_xdma_201920_1.xclbin'
TEST PASSED

実行後に，dmesg をみてみると，

[ 2797.275357] xocl 0000:01:00.1: _xocl_drvinst_open: OPEN 1
[ 2797.275359] [drm] creating scheduler client for pid(9823), ret: 0
[ 2797.276355] xmc.u xmc.u.11534336: xmc_read_from_peer: reading from peer
[ 2797.276387] mailbox.u mailbox.u.13631488: mailbox_request: sending request: 10 via HW
[ 2797.276585] mailbox.m mailbox.m.13631488: process_request: received request from peer: 10, passed on
[ 2797.276602] xclmgmt 0000:01:00.0: xclmgmt_read_subdev_req: req kind 0
[ 2797.276694] mailbox.m mailbox.m.13631488: mailbox_post_response: posting response for: 10 via HW
[ 2797.278266] icap.u icap.u.15728640: icap_read_from_peer: reading from peer
[ 2797.278268] mailbox.u mailbox.u.13631488: mailbox_request: sending request: 10 via HW
[ 2797.278417] mailbox.m mailbox.m.13631488: process_request: received request from peer: 10, passed on
[ 2797.278418] xclmgmt 0000:01:00.0: xclmgmt_read_subdev_req: req kind 1
[ 2797.283449] mailbox.m mailbox.m.13631488: mailbox_post_response: posting response for: 10 via HW
[ 2797.309228] xocl 0000:01:00.1: xocl_read_axlf_helper: xclbin is already downloaded
[ 2797.309231] xocl 0000:01:00.1: xocl_read_axlf_helper: Loaded xclbin ee9018f1-0307-4399-9475-114e40d0c467
[ 2797.309302] icap.u icap.u.15728640: icap_lock_bitstream: bitstream ee9018f1-0307-4399-9475-114e40d0c467 locked, ref=1
[ 2797.309303] xocl 0000:01:00.1: exec_reset: exec_reset(0) cfg(1)
[ 2797.309305] xocl_mb_sche mb_scheduler.u.4194304: client_ioctl_ctx: CTX add(ee9018f1-0307-4399-9475-114e40d0c467, pid 9823, cu_idx 0xffffffff) = 0, ctx=1
[ 2797.309339] [drm] command scheduler is already configured for this device
[ 2797.309381] icap.u icap.u.15728640: icap_unlock_bitstream: bitstream ee9018f1-0307-4399-9475-114e40d0c467 unlocked, ref=0
[ 2797.309382] xocl 0000:01:00.1: exec_stop: exec_stop(000000005cc8f72a)
[ 2797.309387] xocl_mb_sche mb_scheduler.u.4194304: client_ioctl_ctx: CTX del(ee9018f1-0307-4399-9475-114e40d0c467, pid 9823, cu_idx 0xffffffff) = 0, ctx=0
[ 2797.311933] icap.u icap.u.15728640: icap_lock_bitstream: bitstream ee9018f1-0307-4399-9475-114e40d0c467 locked, ref=1
[ 2797.311934] xocl 0000:01:00.1: exec_reset: exec_reset(0) cfg(1)
[ 2797.311935] xocl_mb_sche mb_scheduler.u.4194304: client_ioctl_ctx: CTX add(ee9018f1-0307-4399-9475-114e40d0c467, pid 9823, cu_idx 0xffffffff) = 0, ctx=1
[ 2797.312379] xocl_mb_sche mb_scheduler.u.4194304: client_ioctl_ctx: CTX add(ee9018f1-0307-4399-9475-114e40d0c467, pid 9823, cu_idx 0x0) = 0, ctx=2
[ 2797.314118] xocl_mb_sche mb_scheduler.u.4194304: client_ioctl_ctx: CTX del(ee9018f1-0307-4399-9475-114e40d0c467, pid 9823, cu_idx 0x0) = 0, ctx=1
[ 2797.314120] icap.u icap.u.15728640: icap_unlock_bitstream: bitstream ee9018f1-0307-4399-9475-114e40d0c467 unlocked, ref=0
[ 2797.314122] xocl 0000:01:00.1: exec_stop: exec_stop(000000005cc8f72a)
[ 2797.314125] xocl_mb_sche mb_scheduler.u.4194304: client_ioctl_ctx: CTX del(ee9018f1-0307-4399-9475-114e40d0c467, pid 9823, cu_idx 0xffffffff) = 0, ctx=0
[ 2797.341755] [drm] client exits pid(9823)
[ 2797.341757] xocl 0000:01:00.1: xocl_drvinst_close: CLOSE 2
[ 2797.341758] xocl 0000:01:00.1: xocl_drvinst_close: NOTIFY 00000000c3f49de0

と，HWを叩いて実行している様子が確認できました．