| Rev |
Age |
Author |
Path |
Log message |
Diff |
| 37472 |
8 d 20 h |
amit |
/trunk/profitmandi-cron/src/main/java/com/smartdukaan/cron/ |
IMEI activation: one snapshotted daily pass per brand, on one thread, with per-brand metrics
Four @Scheduled jobs every 5 minutes become two daily passes. Oppo and realme
share one thread and alternate in 25-imei chunks, so exactly one ChromeDriver is
alive at a time instead of four; vivo keeps its own thread since it is direct
HTTP and does not contend for a browser.
The pass snapshots its pool before any browser starts and walks that list to the
end. It never re-queries, and that is the actual fix. A failed lookup never
reaches dateMap.put, so no row is written, so createTimestamp is not bumped, so
the imei was eligible again on the next tick five minutes later. Measured 29-Aug:
realme issued 4,524 requests against 1,004 distinct imeis -- 4.5 asks each, 78%
of the day's budget spent re-asking -- while oppo, which rarely fails, sat at
1.03. More requests hardened the block, which caused more failures. A pass bounds
that: a failure costs one retry tomorrow, never one in five minutes.
This supersedes the r37447/r37448/r37449 argument about driver count, which was
about the wrong variable. That argument blamed realme's collapse on CPU
contention pushing the captcha render past the element waits. The logs do not
support it: on 29-Aug oppo took ZERO canvas timeouts across all 24 hours on the
same box, same six cores, same driver count, same captcha vendor, load average
0.9 -- including the 15:00-23:00 window in which realme solved nothing at all.
Realme's own canvas wait is 15s against oppo's 8s, so the longer wait is the one
expiring. What realme's timeout rate tracks is its own daily request volume, and
it resets at midnight: 920/day -> 0.3%, 3,467/day -> 28%, 4,524/day -> 75%. That
is realme.com declining to serve the widget.
DAYS=0 is deliberate and is not an off-by-one: the pool filter is
createTimestamp < now().atStartOfDay().minusDays(DAYS), so DAYS=1 measures
against yesterday midnight and silently yields a two-day cadence, which is what
oppo and realme were running.
Sizing measured on prod for a midnight start: oppo 4,133 and realme 2,118 imeis,
11.7h + 8.4h = 20.1 hours of a single thread. It fits with no slack; if the
'pass finished' counts come in short of 'pass starting', the lever is DAYS=1
rather than a second thread.
Observability: ImeiActivationGauges publishes the funnel per brand on
/actuator/prometheus, which alloy already scrapes on this host -- due, churned,
captcha_shown, captcha_solved, answered, dates_found, errors, run_seconds and
last_finish_epoch. Each stage fails differently and says what broke. Rates are
left to PromQL. The stage that matters for health is answered: churned>0 with
answered==0 is precisely the shape of both silent outages this year (oppo wrote
nothing for a week; the vivo captcha solver was dead for 46 days). dates_found is
deliberately NOT a health signal -- when the multi-year backlog drained at the
end of August, yield fell from ~100% to 2-3% on the same day across all three
brands with nothing broken.
Nagios cleanup: the Nagios server and every NRPE daemon are gone, so
WriteToPropertiesFile and the commented-out blocks that fed
nagios-cron.properties are deleted, and NagiosMonitorTasks is renamed
BalanceMonitorTasks for the transport it actually uses. Noted there that nothing
calls it -- there is no @Scheduled entry and no other caller -- which is why both
balance gauges have always read -1. |
|
| 37446 |
12 d 16 h |
amit |
/trunk/ |
Per-brand batch sizes, and stop chrome forking a GPU process it cannot use
maxResults was hardcoded in the shared repository methods, so Oppo and Vivo were
forced to the same secondary batch (10) and all three to the same tertiary (10).
It is now a parameter, set per brand at the call site.
Sizing is arithmetic, from measured IN-BATCH per-imei time. Solving
M * 86400 / (300 + M*t) = needed/day:
brand needed/day t M required set to
Vivo 9,407 0.8s 36 50 clears, ~12,700/day
Realme 1,973 13.4s 10 10 was 5 = ~1,177/day, short
Oppo 4,798 14.6s 88 10 HELD, see below
Correcting an earlier measurement of mine: I reported Vivo at 13.6s per imei and
concluded its backlog could not be cleared. That averaged across the ~300s idle
gaps BETWEEN batches. In-batch it is 0.8s -- Vivo is 17x faster than I said, is
idle ~97% of the time, and 50 clears its pool comfortably. There is no wait in
the Vivo path; it is simply fast.
Oppo is deliberately NOT raised. At 14.6s it would need M=88, which means
20-minute batches and near-permanent chrome sessions. But that 14.6s predates the
retry cap (r37445), which cuts exhausted imeis from 20 attempts to 7 and should
drop it sharply. Re-measure before sizing Oppo, rather than guessing high on a
box with 3GB free.
Also: --disable-gpu, --disable-dev-shm-usage, --disable-software-rasterizer on
both selenium tasks. Headless needs no GPU yet chrome forks a gpu-process per
browser -- 6 were alive across the fleet, pure overhead. No behaviour change.
Batch size does not raise peak concurrency (fixedDelay means one batch per job at
a time, so never more than 4 drivers). It raises DUTY CYCLE, which converts
chrome's footprint from intermittent to sustained. That matters here: tomcat is
9.4GB, available is ~3GB, and the two OOM kills this month both took tomcat.
Cron-only deploy. The dao signature change has no callers outside cron. |
|
| 37445 |
12 d 18 h |
amit |
/trunk/profitmandi-cron/src/main/java/com/smartdukaan/cron/scheduled/ |
Oppo/Realme: cap captcha retries per tick at 7 and 12, measured per brand
Most captcha work goes to imeis that never succeed. Measured over the last
window: oppo 6,245 attempts for 308 successes, realme 13,842 for 399. The bulk
is imeis grinding all 20 attempts in one already-refused session and getting
nothing -- oppo 279 of them, realme 610.
The imei is NOT abandoned. A captcha failure is a technical failure, not an
answer, so it stays pending and the next tick retries it in ~5 minutes with a
fresh page and session -- which the data suggests beats continuing in a session
the widget has refused seven times. This caps grinding, not retrying.
Per-brand caps, because the distributions genuinely differ:
OPPO cap 7 97.7% of successes kept, 58.4% of work saved (3,648 attempts)
Pre-glide, 33% of oppo's successes came from attempts 8-20 and 20 was the
right number. Post-glide (r37440) that is 2%: the drag lands first or second
try. cap 5 keeps 95.5%/saves 67.7%, cap 10 keeps 99.0%/saves 44.8%.
REALME cap 12 96.0% kept, 35.7% saved (4,937 attempts)
NOT 7. Realme has no glide, so its successes still spread to attempt 10+ and
a cap of 7 would cost it 16.5%. It also burns more than twice oppo's work, so
the gentler cap still saves more in absolute terms.
Combined: 8,585 of 20,087 attempts saved, ~43% less browser work, for ~3% fewer
successes per tick -- and those imeis come back next tick anyway. Less Chrome
work matters on this box: it co-hosts a 9.4GB tomcat, has 3.5GB free and has been
OOM-killed twice this month.
The two caps are independent constants and must not be synced. If glide is ported
to realme, expect its curve to shift left as oppo's did and its cap can drop to
~7 -- but measure it, do not assume it.
Windows are ~2h (oppo's only ~30 min post-glide), so the right numbers may drift.
Worth re-reading the attempt histogram after a full day. |
|
| 37435 |
13 d 6 h |
amit |
/trunk/profitmandi-cron/src/main/java/com/smartdukaan/cron/scheduled/ |
Oppo/Realme: always release the slider; 79% of presses were left held
Found by following the observation that the dragger repositions itself on click.
clickAndHold() is issued at the top of every attempt, but release() sits inside
the moveByOffset > 0 branch. An attempt that fails circle detection, or produces
an unusable calibration, therefore leaves the mouse button DOWN.
Measured over 21 minutes of production:
Oppo 506 presses, 107 released -> 399 never released = 79%
Realme 247 presses, 58 released -> 189 never released = 77%
Actions is one instance per driver, created OUTSIDE the imei loop, so the stuck
press does not end with the attempt -- it leaks into the next attempt and on into
the next imei. The following clickAndHold then lands on an already-held button,
so the widget never repositions the handle and the drag simply continues from
wherever the pointer was abandoned.
That matches what the logs show. Validated locally against both live widgets:
a clean press repositions the handle +22px on the first pixel of movement (half
the 45px handle: moveToElement centres the pointer, the widget aligns to it), and
the transfer is then linear -- 1.10 piece px per mouse px on oppo, 1.20 on realme.
Production instead shows a jump of +0px on 31 oppo attempts and -34..-41px on
another 39, and oppo starts 13% of attempts with margin-left already >= 30px
(mean 9.2px) against realme's 0% (mean 3.2px). Those are the fingerprints of a
drag resuming rather than starting.
Tracks the press in a flag and releases on every exit: before the retry refresh,
and again after the loop so the widget-not-found break cannot carry a held button
into the next imei. Release is wrapped, so a failure to release is logged rather
than killing the batch.
Ruled out while getting here, so as not to re-chase: the reposition is NOT an
animation race (position is stable from t=0 to 1s despite transition:all), and it
is NOT asymmetric between the brands -- both snap +22px identically, so this bug
does not by itself explain oppo 14% vs realme 64%. Both brands should improve. |
|
| 37433 |
13 d 9 h |
amit |
/trunk/profitmandi-cron/src/main/java/com/smartdukaan/cron/scheduled/ |
Oppo/Realme: split the slider tuning, and give Oppo its own failure verdict
The two classes drive the same dingxiang widget and have been edited in lockstep,
but they do not behave alike: measured over the last 11 hours as successes per
completed drag, oppo runs 11-17% and realme 60%. A change proven on one is not
evidence for the other, so the tuning constants are now documented as per-brand
and must not be synced.
Correcting the record while I am here: oppo has NOT regressed. Earlier "43%" and
"48%" figures were successes over FAILURES, and a small widget-verdict sample --
not successes over drags. On the consistent denominator oppo has been 12-15%
throughout, across r37422, r37424 and r37432 alike.
Oppo: ASSUMED_RATIO 1.7 -> 0.67.
It only sizes the calibration probe: nudge ~= gap / (2 * ratio), intended to
close half the gap. 1.7 came from one hand measurement. Production says
otherwise -- 3,694 oppo calibrations average 0.67 (realme 0.70 over 1,683), so
the probe has been closing about a fifth of the gap, not a half, and the ratio
is then derived from that needlessly small and noisy sample. Mean gap is 120px
for both, and 28% of oppo calibrations already overshoot.
Oppo: log the captcha widget's verdict on the failure branch.
Realme has carried this since r37428 and oppo never did, which is why realme's
failures name POSITION_MISMATCH and oppo's 85% say nothing. Without it there is
no way to tell a bad drag from a widget that never rendered.
Realme: comment only, no behaviour change. Its ratio is wrong in the same way,
but it is wrong while converting 60%, so oppo carries the corrected value first
and realme follows only if it helps. |
|
| 37431 |
13 d 12 h |
amit |
/trunk/profitmandi-cron/src/main/java/com/smartdukaan/cron/scheduled/ |
Realme/Oppo: capture the lookup off Response.prototype, not window.fetch
Realme solves its captcha well -- the widget's own verify endpoint says
success:true on 526 of 829 attempts, 63% -- and records nothing. Every one of
those solved captchas is logged as "Failed" and the imei re-queued.
The lookup is not missing, it is invisible. realme's client calls Nuxt's $fetch
(ofetch), which takes its reference to globalThis.fetch when its bundle is
evaluated. That happens before driver.get() returns, so by the time
installResponseCapture wraps window.fetch the client is already holding the
original. r37425 added that wrapper and could never have worked.
Verified on the live page: the same $fetch request is invisible to a
window.fetch wrapper and visible to a Response.prototype one.
Response.prototype.json/.text are resolved per call, so patching them catches
any client whenever it took its fetch reference -- the same reason the
XMLHttpRequest.prototype.open patch has always worked (it is how the captcha
verdict gets captured today).
The clone is read with the ORIGINAL reader. Calling the patched one recurses
forever and hangs the page; that is not theoretical, it hung a test run.
Applied to Oppo as well. Its XHR path works today (673 lookups, 0 missed), so
this is defence against the same trap following any migration -- which is what
r37425 was reaching for and did not achieve.
Expected: Realme goes from 0 rows to roughly its captcha rate. Unchanged:
59% of attempts still never find the circles, same ceiling as Oppo. |
|
| 37430 |
13 d 16 h |
amit |
/trunk/profitmandi-cron/src/main/java/com/smartdukaan/cron/scheduled/ |
Realme: widen the lookup match and report what the page actually called
The captcha is being solved -- the widget returns success:true with a token --
but the lookup response is never captured, so awaitResponse times out and every
attempt records as failed. 19 'success' against 53 'failed' this hour with 0
lookups, while the widget's own verdicts are mostly success:true.
Matches on the customer-api host as well as the active/check path, in case the
endpoint moved with the page rebuild. A failed attempt now also logs the
non-captcha URLs the page did call, so if it is still missed the log names what
to match on rather than leaving it to guesswork. |
|
| 37428 |
13 d 17 h |
amit |
/trunk/profitmandi-cron/src/main/java/com/smartdukaan/cron/scheduled/ |
Oppo/Realme: record the captcha widget's own verdict alongside the lookup
r37426 made the lookup response the success signal, which is authoritative but
silent about WHY a drag was refused -- a timeout looks identical whether the aim
was wrong, the widget never rendered, or the page changed again.
The captcha widget answers that itself: it calls captcha-ind-sec.heytapmobile.com
and its replies carry a status and message. The hook now keeps the last such
response, and a failed attempt logs it, so 'Failed = 3' becomes 'Failed = 3
(captcha said: ...)'.
Note what the widget does NOT give us: its init call returns the puzzle's y
coordinate but not the x, which is the secret being protected. So the circle
detection still has to find the horizontal target -- there is no shortcut there,
and that remains the largest loss (60% of Oppo attempts never find the circles). |
|
| 37426 |
13 d 17 h |
amit |
/trunk/profitmandi-cron/src/main/java/com/smartdukaan/cron/scheduled/ |
Realme: treat the lookup response, not a missing error element, as success
After r37425 Realme logged 27 'Success at attempt' and 27 'no active/check
response captured' in the same window, and wrote 0 rows. The hook was not at
fault -- reproduced locally, it captures both XHR and fetch correctly.
Success was being inferred from the ABSENCE of dx_captcha_basic_bar-inform:
try { wait5Sec.until(visibilityOf(bar-inform)); log('Failed') }
catch { log('Success') }
realme rebuilt this page, so that element may no longer exist. The wait then
always times out and EVERY attempt reports success while the captcha was never
solved, no lookup fired and nothing was captured. The 27/27/0 is that exactly.
The response arriving is the only signal that actually proves the captcha
passed, so it now waits for that (polling, up to 8s) and treats its absence as a
failure worth retrying. This also removes the separate 'no response captured'
branch, since reaching the success path now means the response is in hand.
Oppo is left alone: its bar-inform detection is working (0 missed captures
against 673 lookups), so there is nothing to fix there. |
|
| 37425 |
13 d 17 h |
amit |
/trunk/profitmandi-cron/src/main/java/com/smartdukaan/cron/scheduled/ |
Oppo/Realme: capture the response from fetch as well as XHR
r37423 got Realme solving the captcha again -- 4 of 4 on the first tick after
deploy -- but every one logged 'Result shown but no active/check response
captured'. The capture hook only wrapped XMLHttpRequest, and realme's page issues
the lookup through fetch, so it recorded nothing and each imei was stored with a
null date.
The hook now wraps window.fetch too, cloning the response before reading it so
the page still consumes its own body. Applied to Oppo as well: its XHR path is
working today, but the same silent failure would follow any move to fetch, and
the cost is a few lines. |
|
| 37424 |
13 d 18 h |
amit |
/trunk/profitmandi-cron/src/main/java/com/smartdukaan/cron/scheduled/ |
Oppo/Realme slider captcha: size the calibration nudge from the gap
r37422 replaced a 15px calibration nudge with a fixed 60px, which fixed the
under-measurement but introduced the opposite error. Production data over 764
calibrations shows 45% ended UNUSABLE (closed <= 0, i.e. the piece shot past the
hole and the measured gap grew instead of shrinking), and among gaps under 60px
it was 85%.
The nudge is now sized from the gap -- aim to close about half of it -- and
clamped to [10,60]. Replaying the real observed gap distribution: unusable and
overshoot fall from 21% to 2%.
The observed travel ratio in production is 1.13 circle px per slider px, against
1.68 measured locally; it varies with render scale, which is why the ratio is
still derived from the measurement. ASSUMED_RATIO only sizes the probe, and 1.7
errs toward a smaller nudge, which is the safe direction here.
Oppo is currently solving 43% (was 9% before r37422). Realme carries the same
widget and the same defect and has not been deployed yet. |
|
| 37423 |
13 d 18 h |
amit |
/trunk/profitmandi-cron/src/main/java/com/smartdukaan/cron/scheduled/ |
Realme warranty check: retarget the rebuilt page and read its JSON
Realme has recorded nothing since 24-Aug, the same day Oppo stopped. Driving the
page confirms why: it was rebuilt and every selector the code used is gone.
.sn-input -> the field is now .imei-input input (placeholder 'Please input IMEI')
div.check-btn -> the submit is now an icon, .imei-input .search-icon
So it failed even earlier than Oppo did -- it could not type the imei, let alone
reach the captcha, which is why the log showed only 'Initiating webdriver...'.
The result now comes from the JSON the page already receives
(customer-api.realme.com/in/active/check) rather than an XPath keyed off a label
that no longer exists. That old locator also had an '| //h1[contains(@class,
title)]' branch which would silently match a page heading and try to parse it as
a date. The API is explicit: expiryDate (yyyy.MM.dd) and an isActivation flag,
so nothing is inferred from message text.
expiryDate is the WARRANTY EXPIRY, so a year is still subtracted -- warranty is
12 months as standard. Checked against our own records rather than assumed:
41,267 of 41,289 stored Realme activation dates precede the date we checked
them, averaging 13 days before, which is only consistent with the subtraction
already being correct.
Ports the r37422 captcha fix verbatim -- identical dingxiang slider, identical
15px calibration nudge that made it land ~30px out against a ~30px hole. And as
with Oppo the outcome is always recorded, even as null: findElement THREW when
the element was missing, so nothing was written and the imei was re-queued every
5 minutes indefinitely.
Also converts 29 System.out.println to the logger. The cron JVM's stdout is a
socket from the launching ssh session, so every diagnostic was going nowhere. |
|
| 36583 |
114 d 21 h |
amit |
/trunk/profitmandi-cron/src/main/java/com/smartdukaan/cron/scheduled/ |
Fix hardcoded captcha element IDs to dynamic xpath in Oppo and Realme warranty scrapers |
|
| 36306 |
143 d 5 h |
amit |
/trunk/profitmandi-cron/src/main/java/com/smartdukaan/cron/ |
Batch processing: BatchScheduledTasks, helpers for offer/sellin/partnerLimit, CronBatchService, OpenCV fix for Apple Silicon, CLI triggers |
|
| 36253 |
150 d 2 h |
amit |
/trunk/profitmandi-cron/src/main/java/com/smartdukaan/cron/scheduled/ |
Separate secondary/tertiary IMEI activation crons for Vivo/Oppo/Realme, perf fixes: shared saveActivation, Response leak fixes, /tmp cleanup, OpenCV static init, early break, remove class-level @Transactional from StandAlone |
|
| 34681 |
445 d 21 h |
amit.gupta |
/trunk/profitmandi-cron/src/main/java/com/smartdukaan/cron/scheduled/ |
Fixed |
|
| 34679 |
445 d 21 h |
amit.gupta |
/trunk/profitmandi-cron/src/main/java/com/smartdukaan/cron/scheduled/ |
Fixed |
|
| 34424 |
511 d 19 h |
amit.gupta |
/trunk/profitmandi-cron/src/main/java/com/smartdukaan/cron/scheduled/ |
Fixed Check Realme Warranty Task |
|
| 34420 |
512 d 17 h |
amit.gupta |
/trunk/ |
Added Realme Activation capture logic |
|
| 34419 |
512 d 19 h |
amit.gupta |
/trunk/profitmandi-cron/src/main/java/com/smartdukaan/cron/scheduled/ |
config |
|