| Rev |
Age |
Author |
Path |
Log message |
Diff |
| 37788 |
13 d 7 h |
amit |
/trunk/profitmandi-cron/ |
activation: stop a broken JVM from spending the day's imei pool, and take the
selenium atom read out of the nested jar
Oppo/Realme activation collapsed on 21-22 Sep. Every WebDriver command failed with
java.util.zip.ZipException reading a Selenium JS atom:
W3CHttpCommandCodec.amendParameters:227 -> executeAtom:397
-> com.google.common.io.Resources.toString
-> org.springframework.boot.loader.jar.ZipInflaterInputStream.read -> ZipException
Scale, from fofo.activated_imei: a ~30-50 errors/day baseline became 17,508 on 21-Sep
and 27,768 on 22-Sep. On 22-Sep it touched 3,920 realme imeis for 0 dates and 4,824
oppo for 96, against a normal 76-100% hit rate. Realme burned its entire day pool by
11:07 and then correctly went quiet, having answered nothing.
TWO INDEPENDENT FAULTS, one fixed each way.
1. The pool was spent on an outage. restUnanswered stamps every unanswered imei so the
20-second tick advances instead of re-handing the same rows -- correct for a per-imei
failure, catastrophic for a systemic one, because a stamped row does not come back
until its rest expires. So a JVM that cannot read a jar quietly consumed a day of
payout data. Now: if NOTHING in the chunk was answered and the chunk had more than one
imei, that is infrastructure rather than a verdict, and nothing is stamped. Cost is a
re-ask of the same chunk next tick -- loud and self-limiting -- instead of the day.
Same class of bug as the carlcare transient refusal (r37537): far-end/our-end noise
must never be recorded as an answer.
2. The read itself. Selenium loads its atoms as classpath RESOURCES on essentially
every command, which inside a fat jar is a nested-jar read on the Spring Boot 2.0.2
(2018) loader. Three different inflater errors appeared on prod -- "invalid stored
block lengths", "invalid distance too far back", "invalid code lengths set" -- on a jar
whose outer AND extracted nested archives both pass `unzip -t`. Intact bytes with three
distinct inflater failures is a reader fault, not a file fault. bootJar now sets
requiresUnpack for selenium-remote-driver, so it is extracted to a real file at launch
and the atom read never touches the nested reader. Verified in the built jar: the entry
carries UNPACK:<sha1> and is STORED rather than DEFLATED.
TRAPS WORTH RECORDING.
- A restart is NOT a diagnosis here. It was restarted 13:16 on 22-Sep and still failed
for three more hours at ~1,260/hour, then a 16:36 restart came up clean -- same jar,
mtime unchanged. Anyone reading "restart fixed it" should distrust it.
- Concurrency alone does not explain it: up to 2 scheduler pools ran these tasks per
minute in the broken window AND in the healthy one.
- The GlitchTip board under-reported this badly (#1495 lastSeen 17-Sep while the log
held thousands on 21-22 Sep), so the board is not a reliable outage signal for cron.
Judge this lane on dates written in fofo.activated_imei, not on issue counts.
- Deploy the cron jar stop -> replace -> start. The jar on disk was overwritten in place
at 12:31 on 21-Sep while a JVM held it open, which is how this started. |
|
| 37565 |
30 d 20 h |
amit |
/trunk/profitmandi-cron/src/main/java/com/smartdukaan/cron/ |
IMEI activation: 20s tick lanes for oppo/realme/vivo, idle once the day clears
Replaces the two once-a-day passes with chunk-per-tick lanes. Each tick takes one
chunk and returns; when a brand's pool comes back empty its turn is skipped, and
when everything is clear the ticks do nothing and stay silent until midnight.
- browser lane (StandAlone): one chunk of 25 for one brand every 20s, oppo and
realme by turns. Still exactly one ChromeDriver alive at a time.
- vivo lane: its own tick, chunk of 250. It must not share the browser lane --
0.24s an imei against oppo's 10.2s means it would need ~70 hours behind them
for work it does alone in 39 minutes.
The snapshot is gone; the pool query is the cursor. That is what makes a restart
cost one chunk instead of the day: the 12:03 restart on 09-Sep forfeited ~4,000
lookups and the whole afternoon, and last_finish had read -1 for three days.
The snapshot existed to stop the re-ask loop (realme, 29-Aug: 4,524 requests
against 1,004 distinct imeis). That is now closed at the source instead -- oppo,
realme and motorola stamp every imei they asked about, not just the ones that
produced a map entry, so a failed lookup rests until tomorrow rather than coming
back on the next tick. Motorola is fixed pre-emptively; nothing schedules it yet.
Also:
- secondary and tertiary are merged by turns rather than concatenated. Safe while
a pass walked to the end; without that guarantee oppo's 163 tertiary serials sat
behind 3,819 secondary ones and would only be reached on a day that cleared.
- vivo abandons a tick rather than the chunk when the captcha solver returns no
code -- one probe per 20s while it is down instead of 250, and no rows rested
over a transient outage.
- the funnel gauges move from a pass to a day. due is measured on the first tick
after midnight, the rest accumulate, and last_finish_epoch becomes a real
completion clock. Truncation is detected at the midnight rollover, which is the
case that never reaches an end-of-run at all.
This does not create capacity. At the 21s/imei measured on 09-Sep the pool still
needs ~35 hours and will not clear; it now rolls over visibly instead of silently.
The lever for that is DAYS=1. |
|
| 36253 |
181 d 12 h |
amit |
/trunk/profitmandi-cron/src/main/java/com/smartdukaan/cron/scheduled/ |
Separate secondary/tertiary IMEI activation crons for Vivo/Oppo/Realme, perf fixes: shared saveActivation, Response leak fixes, /tmp cleanup, OpenCV static init, early break, remove class-level @Transactional from StandAlone |
|
| 34415 |
544 d 8 h |
amit.gupta |
/trunk/ |
Fixed oppo activation |
|
| 34413 |
545 d 5 h |
amit.gupta |
/trunk/profitmandi-cron/src/main/java/com/smartdukaan/cron/scheduled/ |
Fixed oppo tracking issues |
|
| 30377 |
1622 d 2 h |
amit.gupta |
/trunk/profitmandi-cron/src/main/java/com/smartdukaan/cron/scheduled/ |
Added Einvoice Files |
|
| 30372 |
1622 d 4 h |
amit.gupta |
/trunk/profitmandi-cron/src/main/java/com/smartdukaan/cron/scheduled/ |
Added Einvoice Files |
|
| 30355 |
1626 d 5 h |
amit.gupta |
/trunk/profitmandi-cron/src/main/java/com/smartdukaan/cron/scheduled/ |
Fixed ahead issue |
|
| 30352 |
1626 d 8 h |
amit.gupta |
/trunk/profitmandi-cron/src/main/java/com/smartdukaan/cron/scheduled/ |
Fixed oppo |
|
| 30315 |
1637 d 5 h |
amit.gupta |
/trunk/profitmandi-cron/src/main/java/com/smartdukaan/cron/scheduled/ |
Added Einvoice Files |
|
| 30209 |
1659 d 6 h |
amit.gupta |
/trunk/profitmandi-cron/src/main/java/com/smartdukaan/cron/ |
Added Oppo Activation |
|