The reason was that the code hit the line "canerr("ERROR: No available mailbox\n");" in stm32_cansend, so can_xmit thinks it has sent the packet to the hardware, but actually has not. Therefore the transmit interrupt never happens which would call can_txdone, and so the size of the FIFO size does not decrease.
The reason why the code actually hit the mentioned line above, is because stm32can_txready uses a different (incomplete) condition than stm32can_send to determine if the mailbox can be used for sending, and thus can_xmit forwards the packet to stm32can_send. stm32can_txready considered mailboxes OK for sending if the mailbox was empty, but did not consider that mailboxes may not yet be used if the request completed bit is set - stm32can_txinterrupt has to process these mailboxes first.
Note that I have also modified stm32can_txinterrupt - I removed the if condition, because the CAN controller retries to send the packet until it succeeds. Also if the condition would not evaluate to true, can_txdone would not be called and the FIFO size would not decrease also.
b8e7d5c455