Industries · Trading and HFT
For matching engines and feed handlers
ByteAsk joins the gdb session you already have open, finds the lock-order inversion, and cites ITCH and FIX by section.
The deadlock is found and fixed.
❯ byteask "my gdb is stopped on the deadlock in orderd, take a look"
● Joined the bridge in your gdb. orderd is stopped.
▸ Two threads are stuck in __lll_lock_wait with no progress since 09:31.
▸ T7 (match) is blocked in OrderBook::cancel at order_book.cpp:214.
▸ T7 holds book_mtx_ and waits on risk_mtx_, owned by LWP 71154.
▸ T12 holds risk_mtx_ and waits on book_mtx_, owned by LWP 71149.
▸ That is a lock-order inversion between cancel() and on_fill().
✎ order_book.cpp
-std::lock_guard<std::mutex> bk(book_mtx_);
-std::lock_guard<std::mutex> rk(risk_mtx_);
+std::scoped_lock lk(book_mtx_, risk_mtx_);
▸ The same change is applied in risk_engine.cpp:87.
$ ninja -C build && ./build/stress --orders 5000000 --threads 16
✓ 5000000 orders, 0 stalls
▸ I want to run detach in your gdb. Approve? [y/n]
▸ Approved. Sent to your gdb.
✓The diff is ready for review.
(gdb) source ~/.byteask/byteask_gdb_bridge.py your session · 09:36
(gdb) byteask-bridge
ByteAsk bridge armed. The AI can attach now.
[byteask] (gdb) info threads
Id Target Id Frame
* 1 LWP 71142 "orderd" epoll_wait ()
2 LWP 71143 "io" io_uring_enter ()
7 LWP 71149 "match" __lll_lock_wait ()
9 LWP 71151 "md_feed" recvmsg ()
12 LWP 71154 "risk" __lll_lock_wait ()
[byteask] (gdb) thread 7
[Switching to thread 7 (LWP 71149 "match")]
[byteask] (gdb) bt
#0 __lll_lock_wait ()
#1 ___pthread_mutex_lock ()
#2 std::mutex::lock () at bits/std_mutex.h
#3 OrderBook::cancel (this=0x55d1a3f2c8e0, id=88213004) at order_book.cpp:214
[byteask] (gdb) frame 3
#3 OrderBook::cancel (this=0x55d1a3f2c8e0, id=88213004) at order_book.cpp:214
[byteask] (gdb) p risk_mtx_._M_mutex.__data.__owner
$1 = 71154 ← held by T12
[byteask] (gdb) thread 12
[Switching to thread 12 (LWP 71154 "risk")]
[byteask] (gdb) bt
#0 __lll_lock_wait ()
#1 ___pthread_mutex_lock ()
#2 std::mutex::lock () at bits/std_mutex.h
#3 RiskEngine::on_fill (this=0x55d1a3f30120, fill=...) at risk_engine.cpp:87
[byteask] (gdb) p book_mtx_._M_mutex.__data.__owner
$2 = 71149 ← held by T7 · cycle confirmed
[byteask] (gdb) detach
Detaching from program: /srv/orderd, process 71142
[Inferior 1 (process 71142) detached]
(gdb)
Recorded ByteAsk session on a trading service stopped in gdb: the orderd deadlock.
Standards it can cite
ITCH/OUCH · FIX · PTP
hft_ref: ITCH/OUCH · FIX · IEEE 1588 PTP
Nasdaq TotalView-ITCH 5.0 Specification§1.3 Add Order Message · p.13
Add Order Message
Name Offset Length Value Notes
Message Type 0 1 “A” Add Order – No MPID Attribution Message.
Stock Locate 1 2 Integer Locate code identifying the security
Tracking Number 3 2 Integer Nasdaq internal tracking number
Timestamp 5 6 Integer Nanoseconds since midnight.
Order Reference 11 8 Integer The unique reference number assigned to the new order at
Number the time of receipt.
Buy/Sell Indicator 19 1 Alpha The type of order being added.
“B” = Buy Order. “S” = Sell Order.
Shares 20 4 Integer The total number of shares associated with the order being
added to the book.
Stock 24 8 Alpha Stock symbol, right padded with spacesThe tools it runs for Trading and HFT
- gdb
- ThreadSanitizer
- perf
- Valgrind
- CMake
- clang-tidy
Enterprise
See Enterprise Your code stays inside your walls
On-prem, air-gapped and bring-your-own-key deployments, with the controls your security team will ask for.
