(root)/ – Rev 37569
Rev 37568 |
Last modification |
Compare with Previous |
View Log
| RSS feed
Last modification
- Rev 37569 2026-09-10 12:05:44
- Author: amit
- Log message:
- fcm: stop holding a row lock on the whole batch across every FCM call
sendNotification lived in ScheduledTasks, which carries a class-level @Transactional,
so one pass was one transaction: it loaded EVERY pending row uncapped, then per row
made an FCM call over the network and stamped sentTimestamp on the entity. A dirty
row stays write-locked until commit, so a run held a lock on every notification in
the batch for the SUM of all the remote waits. Volume is real -- 11,581 sent on
2026-09-09, 12,036 on 2026-09-05 -- and HttpClientFactory sets a 10s socket timeout,
so one bad run could hold thousands of locks for minutes.
That is the shape behind a 33 SECOND average InnoDB row-lock wait server-wide
(Innodb_row_lock_time 63,291s over 15.4 days uptime, max 51,941ms against a 50s
innodb_lock_wait_timeout -- which is where LockAcquisitionException comes from).
Moved to PushNotificationSendService using the same seam the invoicing pass already
uses in InvoiceService.updateIrnsToInvoices: one short read transaction to pick up
the batch and build the payloads, NO transaction across the network call, and one
REQUIRES_NEW transaction per row to record the outcome. No lock is held while waiting
on FCM.
It is a separate bean, and reaches its own transactional methods through an injected
self-reference, because REQUIRES_NEW is applied by a Spring proxy and a plain
self-invocation would bypass it -- same idiom, for the same reason, as InvoiceService.
Also caps the batch at 500/run (30,000/hour against a 12,000 peak day) using the
already-present but never-wired selectPendingNotifications(limit); the previous query
was uncapped, which is what let a campaign burst become one multi-minute transaction.
Send outcomes are unchanged: 200 stamps now, anything else stamps the 1970 sentinel.
The HttpClient is now closed, which it previously was not.