| Rev |
Age |
Author |
Path |
Log message |
Diff |
| 37461 |
12 h 58 m |
ranu |
/trunk/profitmandi-dao/src/main/java/com/spice/profitmandi/dao/model/ |
rbm drr dashboard new ui looks |
|
| 37460 |
13 h 2 m |
ranu |
/trunk/ |
rbm drr dashboard new ui looks |
|
| 37459 |
14 h 46 m |
amit |
/trunk/profitmandi-cron/src/main/java/com/smartdukaan/cron/scheduled/ |
Motorola IMEI activation: secondary + tertiary in one browser session
Mirrors the oppo/realme combined jobs. MotorolaImeiActivationService drives
CheckMotorolaWarrantyTask, with MotorolaChallengeSolver for the challenge.
Cadence comes from the pool query, which defers an imei for `days` after each
attempt (saveActivation bumps createTimestamp even when no date came back), so
days=2 retries everything every two days. Pending pool measured 1,534
(1,182 secondary + 352 tertiary); at ~10-14s/imei, 60 per invocation is about
12 minutes of driver time and clearing the pool inside 48h needs roughly 26
invocations, i.e. an OS cron entry every 90 minutes.
Do NOT schedule it inside the oppo/realme window: each driver tree costs
~850MB and this box has been OOM-killed twice with tomcat the victim, so peak
concurrent drivers is the number that matters. |
|
| 37458 |
15 h 54 m |
aman |
/trunk/profitmandi-fofo/src/main/ |
LOI payment screenshot: report upload failures instead of reporting the field as missing
Submit validated the hidden #payment-sc-docId, but a file input keeps displaying the
chosen filename regardless of what happens next - so a failed /document-upload left the
field looking attached to the operator and missing to the validator ("The following
fields are required: Payment screenshot" on a form that was in fact filled in).
The failure was silent three ways: doAjaxUploadRequestHandler had no error handler; the
global ajaxError reporter calls bootbox, which loads from cdnjs and throws if that is
blocked; and the confirm('Confirm file upload ?') gate returned false with no trace once
Chrome's "prevent this page from creating additional dialogs" box was ticked.
- common.js: optional error callback on doAjaxUploadRequestHandler (additive, opts out of
the generic global dialog via suppressGlobalError); uploadErrorMessage() extracts the
server's real reason from the JSON body or the plain-text duplicate-request body;
uploadDocument() pre-checks size/type, verifies a document_id actually came back, and
reports a message naming the file - .HEIC and .webp get specific guidance;
showAlert() falls back to native alert when bootbox is absent.
- loi-form.js: drop the confirm() gate, 'input' -> 'change', and clear the file input
whenever the id does not land so the widget can never claim an unuploaded file. Status
line under the field (Uploading... / Attached: name / Not attached). Submit now
distinguishes not-picked, still-uploading and upload-failed. Same treatment for the
OnboardingDocumentTable and mk_brand-fee-collection-details uploads, and the brand-fee
dialog names the Attachment field instead of echoing the raw form key.
- loi-form.vm, pendingForm.vm: accept="..." on both file inputs plus the status span.
- FileUploadController: reject empty files, report actual size against the actual limit,
and resolveExtension() falls back to the filename when the browser sends a generic MIME
type (Android file managers send application/octet-stream for ordinary JPG/PNG/PDF).
- AppConfig: static resource version 395 -> 396 for the JS change. |
|
| 37457 |
21 h 27 m |
amit |
/trunk/profitmandi-fofo/src/main/ |
Lock the KYC state to the GSTIN on the LOI
The LOI already captures and validates a GSTIN, and the first two characters of a
GSTIN are its GST state code - so once the LOI exists the KYC state is a fact
about that number, not something to re-type. When one is present the state
dropdown is prefilled from it and locked; otherwise it stays a normal
master-backed select.
Derived from the GSTIN rather than loi_form.gstState: that column is typed by
hand and holds values like 'UP', 'UTTAR PARDESH' and 'MADHAY PARDESH' which
resolve against nothing, and it is blank on 3 rows. Checked against prod - all
693 GSTINs on record are 15 characters and every one resolves through
statemaster.stateCode, and in all 14 rows where the two disagree the GSTIN is
right and the typed value is wrong or empty.
The lock is per row, not per page: the KYC screens list many onboardings, so the
derived state is passed as a map keyed by onboarding id. Blank rows carry their
onboarding id so a new KYC entry is locked too, which is the case that matters.
data-gst-locked keeps the row's dblclick edit toggle from re-enabling the field;
the value still submits because the JS reads it explicitly with .val(). |
|
| 37456 |
21 h 34 m |
amit |
/trunk/ |
Block billing when NIC rejects the transporter GSTIN for e-way bills
NIC files the e-way bill alongside the IRN, so a deregistered transporter
GSTIN returns Status=1 with InfoDtls[InfCd=EWBERR] (3029 "GSTIN - ... is
not active"): the IRN is valid while ewb_no stays null. Nothing downstream
reads that as a failure, so invoices kept being issued for goods that
could not legally move.
Cache the rejected transporter GSTIN in Redis and refuse to bill through
it. The block is keyed on the GSTIN, since one GSTIN is shared by several
warehouse_provider rows, and it carries the day it was raised so it lapses
at midnight and each new day re-tests NIC once. Correcting the GSTIN in
the provider panel lifts it immediately.
Only errors that are the transporter's fault block billing - NIC's 3029,
or any message naming the GSTIN we sent as TransId. Every other EWBERR
behaves as before: the IRN is filed and the e-way bill is retried later.
Guard sits in addBillingDetailsForGrouppedOrders before the pessimistic
lock and before any mutation, mirroring LogisticsServiceImpl#getEwbDetails
(order's own warehouse; self-pickup and runner skipped, as they travel on
a vehicle number rather than a transporter id). |
|
| 37455 |
21 h 46 m |
amit |
/trunk/profitmandi-fofo/src/main/ |
Make the KYC state a master-backed dropdown instead of free text
fofo-kyc-row.vm and legal-kyc-row.vm captured state as a free-text input, so
nothing stopped a spelling the master does not hold. That state is resolved back
through inventory.statemaster by name later, and an unresolvable one cannot be
turned into a state code. Both now render a $stateNames select, and all five
render paths in PartnerOnBoardingPanelController supply the list.
panel-listing.js reads these fields positionally, by td index and
input[type='text']. A select does not match that selector, so the value would
have come back undefined and KYC would have saved a blank state silently -
both selectors now target the select. Row editing is unaffected: the enable
toggle uses :input, which covers select.
Bump the asset version for the panel-listing.js change. |
|
| 37454 |
21 h 52 m |
amit |
/trunk/profitmandi-web/src/main/java/com/spice/profitmandi/web/config/ |
Build googleMailSender as AuthenticatedIdentityMailSender
The bean authenticates as sdtech@smartdukaan.com, so Google Workspace only permits
that address in From. Callers hardcode noreply@, which Gmail refuses with 535
'Authenticated user is not authorized to send mail', silently dropping every alert
sent through it. The sender now rewrites From to its own username at send time.
See profitmandi-common r37451. |
|
| 37453 |
21 h 52 m |
amit |
/trunk/profitmandi-cron/src/main/java/com/smartdukaan/cron/ |
Build googleMailSender as AuthenticatedIdentityMailSender
The bean authenticates as sdtech@smartdukaan.com, so Google Workspace only permits
that address in From. Callers hardcode noreply@, which Gmail refuses with 535
'Authenticated user is not authorized to send mail', silently dropping every alert
sent through it. The sender now rewrites From to its own username at send time.
See profitmandi-common r37451. |
|
| 37452 |
21 h 52 m |
amit |
/trunk/profitmandi-dao/src/main/java/com/spice/profitmandi/service/integrations/gstpro/ |
Fix e-invoice IRN failure when the NIC auth token is about to expire
NIC reissues the same auth token until it genuinely expires, so evicting the cache
inside the 5 minute leeway hands back the identical near-dead token. getAuthenticatedHeaders
checked the leeway once, before that re-fetch, and never inspected what came back.
NSUPHR1770 filed with a token that had 12 seconds of life left and the gateway answered
412/GSP752 'eInvoice AuthToken not found or expired'; the invoice filed a minute later,
past the expiry and so on a genuinely new token, succeeded.
getUsableAuthToken re-checks the refetched token and, when it is still inside a 60s
floor, waits out the remainder before minting again. The wait is bounded by that floor
and every caller on this path is a cron/async thread already running against a 60s NIC
timeout with no transaction held open.
Also make the gateway's own failures legible. NIC rejections arrive as
{Status, ErrorDetails}, but failures the ASP raises in front of NIC use
{status_cd, error{error_cd, message}}, which shares no field name with RespPl: Gson
produced an all-null object and the stored reason degraded to the literal
'RespPl{Status=0, Data=null, ErrorDetails=null, InfoDtls=null}', indistinguishable from
NIC rejecting the document. RespGSPErr already modelled that envelope but was a
non-static inner class and so could not be instantiated by Gson; made it static and
copy the code and message into ErrorDetails, which every caller already reads. |
|
| 37451 |
21 h 53 m |
amit |
/trunk/profitmandi-common/src/main/java/com/spice/profitmandi/common/services/ |
Send authenticated mail as the account that actually authenticates
googleMailSender logs in to smtp.gmail.com as sdtech@smartdukaan.com, but callers
build their From as noreply@smartdukaan.com. Google Workspace binds an authenticated
session to one identity and refuses the mismatch with 535 'Authorization failed:
Authenticated user is not authorized to send mail' - the password is accepted, the
message is not. Every alert routed through that sender was therefore lost at the SMTP
layer, the e-invoice failure alert for NSUPHR1770 among them.
AuthenticatedIdentityMailSender rewrites From to the sender's own username at send
time, keeping whatever display name the caller chose. Senders that do not authenticate
are left alone: gmailRelaySender is the IP-authorised relay and has no username, and
sendGridMailSender's username is the literal 'apikey', so both keep noreply@.
Enforced in the sender rather than at the ~40 call sites that build their own
MimeMessage and hardcode noreply@, so it covers the ones added next too. |
|
| 37450 |
22 h 0 m |
amit |
/trunk/profitmandi-cron/src/main/java/com/smartdukaan/cron/scheduled/ |
Oppo: find only the hole, and let a low-contrast hole be found at all
60% of oppo attempts never got past detection -- getMatCircles2 returned null and
the attempt was wasted before any aiming happened. That is a bigger loss than
everything the scheduling work addressed put together.
Two causes, both visible once real puzzles are captured and looked at.
1. param1 is the Canny HIGH threshold, and it was 100. Edges weaker than that are
discarded before circle finding begins. Measured hole-vs-background contrast on
four live puzzles: 98, 103, 28, 117. The 28 -- a pale lilac background -- cannot
produce an edge at 100, so no circle exists to find. Dropped to 50.
2. It insisted on TWO circles whose radii matched within 3px, and returned null
otherwise. So a puzzle where the hole was located perfectly still failed if the
ring was missed, or if the two radii differed by 4px.
The second is the sillier one: the ring is a DOM element,
dx_captcha_basic_sub-slider_, whose exact position aim() already reads via
pieceCentreX(). Detection was re-finding something known exactly, and then
throwing away a good hole because it could not confirm it. Now the ring position
is passed in and hough only has to find one thing.
Lowering param1 without the pairing check would let textured backgrounds (the
sand images especially) supply spurious circles with nothing to reject them, so
the candidate is verified as an actual hole: its core must be at least 15
luminance below the frame mean. Real holes measure 28-117 below, so 15 rejects
noise with margin.
Validated against live puzzles before committing:
ring is at rel 42px on every sample; hole lands at 113-194px, separation
71-152px -- so RING_EXCLUSION_PX = 30 never rejects a real hole
hole contrast 28-117 against MIN_HOLE_DARKNESS = 15
Not validated locally: the hough call itself. opencv 3.4.2-0 ships no osx/arm64
native and every JDK on this machine is arm64, so the detection path cannot run
here. param1 = 50 is reasoned from the measured contrasts, not measured. If the
detection rate does not move, 40 is the next value to try -- the darkness check is
what makes going lower safe.
Watch "Detected N circle(s) but all sat on the ring" and "Rejecting circle at Npx"
in the log: the first means hough found nothing but the ring, the second means the
darkness check is doing its job. |
|
| 37449 |
1 d 8 h |
amit |
/trunk/profitmandi-cron/src/main/java/com/smartdukaan/cron/scheduled/ |
Realme unmerged too: four selenium jobs, one per pool per brand
r37447 merged secondary and tertiary per brand to halve concurrent drivers.
r37448 reverted that for oppo, because serialising cost it more per imei than a
second browser did. This reverts it for realme as well, which ends the merge
experiment entirely.
What the merge actually cost realme is only visible now that oppo is parallel
again: realme's per-imei went 14.2s -> 29s. It did not change behaviour -- it is
simply competing with oppo's two browsers for CPU. At 29s its merged ceiling is
86400/29 = 2,979/day, just under the 3,054/day the daily re-check needs, so no
batch size could have closed the gap. Two parallel pools restore the ~3,900/day
it managed historically at four drivers.
Measured before this change (25 min window):
brand needed/day throughput note
Oppo 8,838 7,661 parallel revert worked, +122%
Realme 3,054 1,843 merged and CPU-starved
Vivo 10,082 16,128 surplus, cannot transfer to another brand
Sizes unchanged: oppo 25 per pool, realme 12 per pool, vivo 50+10.
Honest accounting of the merge: it was my idea, sized on per-brand arithmetic that
ignored contention between brands, and it is now fully reverted. What survives
from that line of work is the part that actually paid -- reaping orphaned browsers
(~1.8GB), the per-brand retry caps, and per-brand maxResults. Peak drivers are
back to 4, which is where they started.
Watch for contention: four browsers is the configuration that produced the 29s
figure for realme in the first place, so oppo may slow from its current 19.7s.
Re-measure both before tuning sizes again. |
|
| 37448 |
1 d 17 h |
amit |
/trunk/profitmandi-cron/src/main/java/com/smartdukaan/cron/scheduled/ |
Daily re-check for all brands; Oppo back to parallel pools
Two changes.
1. Re-check window 4 days (secondary) and 2 days (tertiary) -> 1 day everywhere.
Daily demand becomes the full universe rather than a fraction of it:
Oppo 4,798 + 4,040 = 8,838/day
Vivo 9,407 + 675 = 10,082/day (doing 15,345 -- fine)
Realme 1,973 + 1,081 = 3,054/day
2. Oppo's two pools run in PARALLEL again, reverting the merge in r37447 for that
brand only. Realme stays merged.
The merge was a straight trade of throughput for memory and oppo could not
afford it. Measured over 32 minutes and again over an hour the next morning:
3,555 then 3,456/day against 5,280 before merging. Batch cadence settled at a
very regular ~15.5 min per cycle, so a 30-imei merged batch takes ~10.5 min =
~21s/imei, against the 10.2s it managed unmerged. At 21s the ceiling is
86400/21 = 4,114/day even with zero idle, so no batch size and no shorter
fixedDelay could have reached 8,838. Serialising simply costs more per imei
here than running two browsers does.
Realme keeps the merge: it needs 3,054/day and delivers 2,952 merged, so a
small size bump covers it without a second browser.
Sizes: oppo 25 per pool (2 jobs in parallel), realme 12+12 merged, vivo 50+10
unchanged. Vivo has already cleared its entire secondary backlog -- the pool
reads 0 and both lists come back empty -- which is what the batch of 50 was for.
Cost: oppo goes back to two concurrent drivers, so the fleet is 3 rather than 2,
roughly +700MB. Acceptable against the ~2GB freed today by reaping orphaned
browsers and capping retries, but it is the reason realme was left merged.
Sizes are a starting point, not a final answer: oppo's per-imei time differs
markedly between merged and parallel modes, so re-measure before tuning further. |
|
| 37447 |
2 d 8 h |
amit |
/trunk/profitmandi-cron/src/main/java/com/smartdukaan/cron/scheduled/ |
Selenium: one browser per brand instead of one per pool
Oppo and Realme each ran secondary and tertiary as separate @Scheduled jobs, so
each brand opened two ChromeDrivers at once and the fleet ran four. Each driver
tree costs roughly 850MB. This box co-hosts a 9.4GB tomcat with ~3GB available
and has been OOM-killed twice this month -- tomcat was the victim both times, so
peak concurrency is the thing that matters here.
Combined into one job per brand. Nothing downstream changes: the two pools differ
only in which named query fills them, and both already funnel into the same
updateActivationDate -> checkWarranty -> saveActivation path. They are disjoint by
construction (secondary excludes anything with a FofoLineItem, tertiary is
FofoLineItem-based); distinct() is insurance, not a fix for a known overlap.
Sizing matters, because merging SERIALISES work that used to run in parallel and
keeping the old batch sizes would quietly cost throughput. Measured post-cap at
10.2s/imei (oppo, down from 14.6 after r37445) and 14.2s/imei (realme), solving
M * 86400 / (300 + M*t):
oppo 2 parallel jobs x10 = 4,299/day -> merged 15+15 = 4,277/day (parity)
realme 2 parallel jobs x10 = 3,910/day -> merged 10+10 = 2,959/day (-24%)
Oppo is sized to hold parity because it is already short of its 4,798/day need.
Realme is left at 20 -- it needs 2,243/day, so it can absorb the dip in exchange
for shorter batches and a shorter-lived browser.
What this saves and does not save: total driver-SECONDS are roughly unchanged,
which is the point of resizing. PEAK concurrent drivers halves from 4 to 2.
Also skips starting a browser at all when both pools come back empty -- currently
never true, but it costs nothing and a browser launched to do nothing is pure
waste on this box.
checkOppoImeiStatus/Tertiary and the realme equivalents are left in place for
manual invocation; they are simply no longer scheduled. |
|
| 37446 |
2 d 8 h |
amit |
/trunk/ |
Per-brand batch sizes, and stop chrome forking a GPU process it cannot use
maxResults was hardcoded in the shared repository methods, so Oppo and Vivo were
forced to the same secondary batch (10) and all three to the same tertiary (10).
It is now a parameter, set per brand at the call site.
Sizing is arithmetic, from measured IN-BATCH per-imei time. Solving
M * 86400 / (300 + M*t) = needed/day:
brand needed/day t M required set to
Vivo 9,407 0.8s 36 50 clears, ~12,700/day
Realme 1,973 13.4s 10 10 was 5 = ~1,177/day, short
Oppo 4,798 14.6s 88 10 HELD, see below
Correcting an earlier measurement of mine: I reported Vivo at 13.6s per imei and
concluded its backlog could not be cleared. That averaged across the ~300s idle
gaps BETWEEN batches. In-batch it is 0.8s -- Vivo is 17x faster than I said, is
idle ~97% of the time, and 50 clears its pool comfortably. There is no wait in
the Vivo path; it is simply fast.
Oppo is deliberately NOT raised. At 14.6s it would need M=88, which means
20-minute batches and near-permanent chrome sessions. But that 14.6s predates the
retry cap (r37445), which cuts exhausted imeis from 20 attempts to 7 and should
drop it sharply. Re-measure before sizing Oppo, rather than guessing high on a
box with 3GB free.
Also: --disable-gpu, --disable-dev-shm-usage, --disable-software-rasterizer on
both selenium tasks. Headless needs no GPU yet chrome forks a gpu-process per
browser -- 6 were alive across the fleet, pure overhead. No behaviour change.
Batch size does not raise peak concurrency (fixedDelay means one batch per job at
a time, so never more than 4 drivers). It raises DUTY CYCLE, which converts
chrome's footprint from intermittent to sustained. That matters here: tomcat is
9.4GB, available is ~3GB, and the two OOM kills this month both took tomcat.
Cron-only deploy. The dao signature change has no callers outside cron. |
|
| 37445 |
2 d 10 h |
amit |
/trunk/profitmandi-cron/src/main/java/com/smartdukaan/cron/scheduled/ |
Oppo/Realme: cap captcha retries per tick at 7 and 12, measured per brand
Most captcha work goes to imeis that never succeed. Measured over the last
window: oppo 6,245 attempts for 308 successes, realme 13,842 for 399. The bulk
is imeis grinding all 20 attempts in one already-refused session and getting
nothing -- oppo 279 of them, realme 610.
The imei is NOT abandoned. A captcha failure is a technical failure, not an
answer, so it stays pending and the next tick retries it in ~5 minutes with a
fresh page and session -- which the data suggests beats continuing in a session
the widget has refused seven times. This caps grinding, not retrying.
Per-brand caps, because the distributions genuinely differ:
OPPO cap 7 97.7% of successes kept, 58.4% of work saved (3,648 attempts)
Pre-glide, 33% of oppo's successes came from attempts 8-20 and 20 was the
right number. Post-glide (r37440) that is 2%: the drag lands first or second
try. cap 5 keeps 95.5%/saves 67.7%, cap 10 keeps 99.0%/saves 44.8%.
REALME cap 12 96.0% kept, 35.7% saved (4,937 attempts)
NOT 7. Realme has no glide, so its successes still spread to attempt 10+ and
a cap of 7 would cost it 16.5%. It also burns more than twice oppo's work, so
the gentler cap still saves more in absolute terms.
Combined: 8,585 of 20,087 attempts saved, ~43% less browser work, for ~3% fewer
successes per tick -- and those imeis come back next tick anyway. Less Chrome
work matters on this box: it co-hosts a 9.4GB tomcat, has 3.5GB free and has been
OOM-killed twice this month.
The two caps are independent constants and must not be synced. If glide is ported
to realme, expect its curve to shift left as oppo's did and its cap can drop to
~7 -- but measure it, do not assume it.
Windows are ~2h (oppo's only ~30 min post-glide), so the right numbers may drift.
Worth re-reading the attempt histogram after a full day. |
|
| 37444 |
2 d 11 h |
ranu |
/trunk/profitmandi-fofo/src/main/webapp/WEB-INF/views/ftl/ |
rbm drr dashboard new ui looks |
|
| 37443 |
2 d 11 h |
ranu |
/trunk/profitmandi-dao/src/main/java/com/spice/profitmandi/dao/model/ |
rbm drr dashboard new ui looks |
|
| 37442 |
2 d 11 h |
ranu |
/trunk/ |
rbm drr dashboard new ui looks |
|