| Rev |
Age |
Author |
Path |
Log message |
Diff |
| 37459 |
39 d 17 h |
amit |
/trunk/profitmandi-cron/src/main/java/com/smartdukaan/cron/scheduled/ |
Motorola IMEI activation: secondary + tertiary in one browser session
Mirrors the oppo/realme combined jobs. MotorolaImeiActivationService drives
CheckMotorolaWarrantyTask, with MotorolaChallengeSolver for the challenge.
Cadence comes from the pool query, which defers an imei for `days` after each
attempt (saveActivation bumps createTimestamp even when no date came back), so
days=2 retries everything every two days. Pending pool measured 1,534
(1,182 secondary + 352 tertiary); at ~10-14s/imei, 60 per invocation is about
12 minutes of driver time and clearing the pool inside 48h needs roughly 26
invocations, i.e. an OS cron entry every 90 minutes.
Do NOT schedule it inside the oppo/realme window: each driver tree costs
~850MB and this box has been OOM-killed twice with tomcat the victim, so peak
concurrent drivers is the number that matters. |
|
| 37448 |
40 d 20 h |
amit |
/trunk/profitmandi-cron/src/main/java/com/smartdukaan/cron/scheduled/ |
Daily re-check for all brands; Oppo back to parallel pools
Two changes.
1. Re-check window 4 days (secondary) and 2 days (tertiary) -> 1 day everywhere.
Daily demand becomes the full universe rather than a fraction of it:
Oppo 4,798 + 4,040 = 8,838/day
Vivo 9,407 + 675 = 10,082/day (doing 15,345 -- fine)
Realme 1,973 + 1,081 = 3,054/day
2. Oppo's two pools run in PARALLEL again, reverting the merge in r37447 for that
brand only. Realme stays merged.
The merge was a straight trade of throughput for memory and oppo could not
afford it. Measured over 32 minutes and again over an hour the next morning:
3,555 then 3,456/day against 5,280 before merging. Batch cadence settled at a
very regular ~15.5 min per cycle, so a 30-imei merged batch takes ~10.5 min =
~21s/imei, against the 10.2s it managed unmerged. At 21s the ceiling is
86400/21 = 4,114/day even with zero idle, so no batch size and no shorter
fixedDelay could have reached 8,838. Serialising simply costs more per imei
here than running two browsers does.
Realme keeps the merge: it needs 3,054/day and delivers 2,952 merged, so a
small size bump covers it without a second browser.
Sizes: oppo 25 per pool (2 jobs in parallel), realme 12+12 merged, vivo 50+10
unchanged. Vivo has already cleared its entire secondary backlog -- the pool
reads 0 and both lists come back empty -- which is what the batch of 50 was for.
Cost: oppo goes back to two concurrent drivers, so the fleet is 3 rather than 2,
roughly +700MB. Acceptable against the ~2GB freed today by reaping orphaned
browsers and capping retries, but it is the reason realme was left merged.
Sizes are a starting point, not a final answer: oppo's per-imei time differs
markedly between merged and parallel modes, so re-measure before tuning further. |
|
| 37447 |
41 d 11 h |
amit |
/trunk/profitmandi-cron/src/main/java/com/smartdukaan/cron/scheduled/ |
Selenium: one browser per brand instead of one per pool
Oppo and Realme each ran secondary and tertiary as separate @Scheduled jobs, so
each brand opened two ChromeDrivers at once and the fleet ran four. Each driver
tree costs roughly 850MB. This box co-hosts a 9.4GB tomcat with ~3GB available
and has been OOM-killed twice this month -- tomcat was the victim both times, so
peak concurrency is the thing that matters here.
Combined into one job per brand. Nothing downstream changes: the two pools differ
only in which named query fills them, and both already funnel into the same
updateActivationDate -> checkWarranty -> saveActivation path. They are disjoint by
construction (secondary excludes anything with a FofoLineItem, tertiary is
FofoLineItem-based); distinct() is insurance, not a fix for a known overlap.
Sizing matters, because merging SERIALISES work that used to run in parallel and
keeping the old batch sizes would quietly cost throughput. Measured post-cap at
10.2s/imei (oppo, down from 14.6 after r37445) and 14.2s/imei (realme), solving
M * 86400 / (300 + M*t):
oppo 2 parallel jobs x10 = 4,299/day -> merged 15+15 = 4,277/day (parity)
realme 2 parallel jobs x10 = 3,910/day -> merged 10+10 = 2,959/day (-24%)
Oppo is sized to hold parity because it is already short of its 4,798/day need.
Realme is left at 20 -- it needs 2,243/day, so it can absorb the dip in exchange
for shorter batches and a shorter-lived browser.
What this saves and does not save: total driver-SECONDS are roughly unchanged,
which is the point of resizing. PEAK concurrent drivers halves from 4 to 2.
Also skips starting a browser at all when both pools come back empty -- currently
never true, but it costs nothing and a browser launched to do nothing is pure
waste on this box.
checkOppoImeiStatus/Tertiary and the realme equivalents are left in place for
manual invocation; they are simply no longer scheduled. |
|
| 37446 |
41 d 11 h |
amit |
/trunk/ |
Per-brand batch sizes, and stop chrome forking a GPU process it cannot use
maxResults was hardcoded in the shared repository methods, so Oppo and Vivo were
forced to the same secondary batch (10) and all three to the same tertiary (10).
It is now a parameter, set per brand at the call site.
Sizing is arithmetic, from measured IN-BATCH per-imei time. Solving
M * 86400 / (300 + M*t) = needed/day:
brand needed/day t M required set to
Vivo 9,407 0.8s 36 50 clears, ~12,700/day
Realme 1,973 13.4s 10 10 was 5 = ~1,177/day, short
Oppo 4,798 14.6s 88 10 HELD, see below
Correcting an earlier measurement of mine: I reported Vivo at 13.6s per imei and
concluded its backlog could not be cleared. That averaged across the ~300s idle
gaps BETWEEN batches. In-batch it is 0.8s -- Vivo is 17x faster than I said, is
idle ~97% of the time, and 50 clears its pool comfortably. There is no wait in
the Vivo path; it is simply fast.
Oppo is deliberately NOT raised. At 14.6s it would need M=88, which means
20-minute batches and near-permanent chrome sessions. But that 14.6s predates the
retry cap (r37445), which cuts exhausted imeis from 20 attempts to 7 and should
drop it sharply. Re-measure before sizing Oppo, rather than guessing high on a
box with 3GB free.
Also: --disable-gpu, --disable-dev-shm-usage, --disable-software-rasterizer on
both selenium tasks. Headless needs no GPU yet chrome forks a gpu-process per
browser -- 6 were alive across the fleet, pure overhead. No behaviour change.
Batch size does not raise peak concurrency (fixedDelay means one batch per job at
a time, so never more than 4 drivers). It raises DUTY CYCLE, which converts
chrome's footprint from intermittent to sustained. That matters here: tomcat is
9.4GB, available is ~3GB, and the two OOM kills this month both took tomcat.
Cron-only deploy. The dao signature change has no callers outside cron. |
|
| 36580 |
143 d 16 h |
amit |
/trunk/profitmandi-cron/src/main/java/com/smartdukaan/cron/scheduled/ |
Adjust IMEI activation deferral: secondary 4 days, tertiary 2 days for Vivo/Oppo/Realme |
|
| 36253 |
178 d 21 h |
amit |
/trunk/profitmandi-cron/src/main/java/com/smartdukaan/cron/scheduled/ |
Separate secondary/tertiary IMEI activation crons for Vivo/Oppo/Realme, perf fixes: shared saveActivation, Response leak fixes, /tmp cleanup, OpenCV static init, early break, remove class-level @Transactional from StandAlone |
|
| 34420 |
541 d 13 h |
amit.gupta |
/trunk/ |
Added Realme Activation capture logic |
|
| 34418 |
541 d 14 h |
amit.gupta |
/trunk/profitmandi-cron/src/main/java/com/smartdukaan/cron/scheduled/ |
config |
|
| 30937 |
1489 d 16 h |
amit.gupta |
/trunk/ |
Fixed activation logic |
|
| 30430 |
1603 d 21 h |
tejbeer |
/trunk/ |
change |
|
| 30353 |
1623 d 15 h |
amit.gupta |
/trunk/profitmandi-cron/src/main/java/com/smartdukaan/cron/scheduled/ |
Fixed ahead issue |
|
| 30352 |
1623 d 17 h |
amit.gupta |
/trunk/profitmandi-cron/src/main/java/com/smartdukaan/cron/scheduled/ |
Fixed oppo |
|
| 30315 |
1634 d 14 h |
amit.gupta |
/trunk/profitmandi-cron/src/main/java/com/smartdukaan/cron/scheduled/ |
Added Einvoice Files |
|
| 30308 |
1637 d 12 h |
amit.gupta |
/trunk/profitmandi-cron/src/main/java/com/smartdukaan/cron/ |
Added Einvoice Files |
|