A tick-to-trade path is measured at 4.2 microseconds. The budget was: 250 ns NIC receive, 100 ns PCIe, 300 ns decode, 400 ns strategy, 300 ns encode, 250 ns NIC send. Where did the missing time go if the process runs on a normal socket with the default scheduler? Account for it with numbers.
A tick-to-trade path is measured at 4.2 microseconds. The budget was: 250 ns NIC receive, 100 ns PCIe, 300 ns decode, 400 ns strategy, 300 ns encode, 250 ns NIC send. Where did the missing time go if the process runs on a normal socket with the default scheduler? Account for it with numbers.
Approach: Add the budgeted stages, subtract from the measured figure, then attribute the remainder to the kernel network stack, the syscall boundary and any scheduler event, each with its own known cost.
About 2.6 microseconds went to the kernel, and a socket path cannot deliver the budget. The stages sum to 1600 ns, so 2600 ns is unaccounted. A read and a write syscall cost roughly 100 ns each in raw entry and exit, but the kernel network stack around them costs far more: interrupt delivery and softirq processing run 1 to 2 microseconds, the skb copy into user memory another few hundred ns, and the wakeup of the blocked thread is a scheduler event that adds 1 to 3 microseconds if the core was idle or running something else. A context switch alone is 1 to 5 microseconds once the cache and TLB working set is disturbed. Kernel bypass removes the interrupt, the softirq, the copy and the wakeup by mapping the NIC ring into the process and busy polling it, which recovers essentially all 2600 ns. The residual after bypass is PCIe and the NIC itself, which is why the budget was written the way it was. Tail latency is worse than this mean because the scheduler event is bimodal.
Follow-up: If you keep the socket path but pin the thread and busy poll with SO_BUSY_POLL, which of those components do you recover and which remain?
Key concepts: kernel bypass, syscall, context switch, tail latency.