Rev 37668 |
Go to most recent revision |
Last modification |
Compare with Previous |
View Log
| RSS feed
Last modification
- Rev 37672 2026-09-17 13:12:56
- Author: amit
- Log message:
- Add BrowserLane: machine-wide interlock so only one headless Chrome runs at a time
Every ChromeDriver tree on the app box costs ~850MB RSS, on a host holding a
-Xmx8g tomcat and a -Xmx2g cron jar on 16GB that has been kernel-OOM-killed
twice with tomcat the victim. Peak concurrent drivers is the number that
matters, not drivers per day.
Measured on the oppo/realme lane: a 25-imei chunk takes 505s (12s driver
startup, then ~19.7s an imei) against a fixed 20s fixedDelay, i.e. a 96% duty
cycle. Meanwhile tomcat starts its own driver 8 times a day for the knowlarity
scrape. On 17-Sep that scrape ran 11:40:00-11:41:07 and the lane's next driver
came up at 11:42:21 -- it cleared by 74 seconds, by luck. At a 96% duty cycle
the collision cannot be dodged by choosing better times, only by serialising.
The two JVMs are separate processes owned by different users, so an in-process
flag cannot see across the boundary. Uses an OS file lock at
/var/lock/sd-browser.lock (overridable via -Dsd.browser.lock), created 0666 on
first use because cron runs as root and tomcat as tomcat and whichever starts
first creates it. A file lock rather than a marker file so the kernel reclaims
it when the holder dies: verified by SIGKILLing the holder mid-hold, the next
JVM acquired in 264ms. A ReentrantLock gates it first, since a second thread in
the same JVM would otherwise get OverlappingFileLockException rather than block.
Callers wait rather than skip. The knowlarity scrape needs 67s against a lane
busy 96% of the time, so skip-on-contention would mean it essentially never
refreshed again. Eight scrapes a day costs the imei lane about nine minutes of
waiting in total.
Fails open if the lock file is unusable (missing /var/lock, read-only mount, dev
laptop): serialising within the JVM only, rather than stopping imei activation
over what is a safety interlock.