| Rev |
Age |
Author |
Path |
Log message |
Diff |
| 37457 |
47 m |
amit |
/trunk/profitmandi-fofo/src/main/ |
Lock the KYC state to the GSTIN on the LOI
The LOI already captures and validates a GSTIN, and the first two characters of a
GSTIN are its GST state code - so once the LOI exists the KYC state is a fact
about that number, not something to re-type. When one is present the state
dropdown is prefilled from it and locked; otherwise it stays a normal
master-backed select.
Derived from the GSTIN rather than loi_form.gstState: that column is typed by
hand and holds values like 'UP', 'UTTAR PARDESH' and 'MADHAY PARDESH' which
resolve against nothing, and it is blank on 3 rows. Checked against prod - all
693 GSTINs on record are 15 characters and every one resolves through
statemaster.stateCode, and in all 14 rows where the two disagree the GSTIN is
right and the typed value is wrong or empty.
The lock is per row, not per page: the KYC screens list many onboardings, so the
derived state is passed as a map keyed by onboarding id. Blank rows carry their
onboarding id so a new KYC entry is locked too, which is the case that matters.
data-gst-locked keeps the row's dblclick edit toggle from re-enabling the field;
the value still submits because the JS reads it explicitly with .val(). |
|
| 37456 |
54 m |
amit |
/trunk/ |
Block billing when NIC rejects the transporter GSTIN for e-way bills
NIC files the e-way bill alongside the IRN, so a deregistered transporter
GSTIN returns Status=1 with InfoDtls[InfCd=EWBERR] (3029 "GSTIN - ... is
not active"): the IRN is valid while ewb_no stays null. Nothing downstream
reads that as a failure, so invoices kept being issued for goods that
could not legally move.
Cache the rejected transporter GSTIN in Redis and refuse to bill through
it. The block is keyed on the GSTIN, since one GSTIN is shared by several
warehouse_provider rows, and it carries the day it was raised so it lapses
at midnight and each new day re-tests NIC once. Correcting the GSTIN in
the provider panel lifts it immediately.
Only errors that are the transporter's fault block billing - NIC's 3029,
or any message naming the GSTIN we sent as TransId. Every other EWBERR
behaves as before: the IRN is filed and the e-way bill is retried later.
Guard sits in addBillingDetailsForGrouppedOrders before the pessimistic
lock and before any mutation, mirroring LogisticsServiceImpl#getEwbDetails
(order's own warehouse; self-pickup and runner skipped, as they travel on
a vehicle number rather than a transporter id). |
|
| 37455 |
1 h 5 m |
amit |
/trunk/profitmandi-fofo/src/main/ |
Make the KYC state a master-backed dropdown instead of free text
fofo-kyc-row.vm and legal-kyc-row.vm captured state as a free-text input, so
nothing stopped a spelling the master does not hold. That state is resolved back
through inventory.statemaster by name later, and an unresolvable one cannot be
turned into a state code. Both now render a $stateNames select, and all five
render paths in PartnerOnBoardingPanelController supply the list.
panel-listing.js reads these fields positionally, by td index and
input[type='text']. A select does not match that selector, so the value would
have come back undefined and KYC would have saved a blank state silently -
both selectors now target the select. Row editing is unaffected: the enable
toggle uses :input, which covers select.
Bump the asset version for the panel-listing.js change. |
|
| 37454 |
1 h 12 m |
amit |
/trunk/profitmandi-web/src/main/java/com/spice/profitmandi/web/config/ |
Build googleMailSender as AuthenticatedIdentityMailSender
The bean authenticates as sdtech@smartdukaan.com, so Google Workspace only permits
that address in From. Callers hardcode noreply@, which Gmail refuses with 535
'Authenticated user is not authorized to send mail', silently dropping every alert
sent through it. The sender now rewrites From to its own username at send time.
See profitmandi-common r37451. |
|
| 37453 |
1 h 12 m |
amit |
/trunk/profitmandi-cron/src/main/java/com/smartdukaan/cron/ |
Build googleMailSender as AuthenticatedIdentityMailSender
The bean authenticates as sdtech@smartdukaan.com, so Google Workspace only permits
that address in From. Callers hardcode noreply@, which Gmail refuses with 535
'Authenticated user is not authorized to send mail', silently dropping every alert
sent through it. The sender now rewrites From to its own username at send time.
See profitmandi-common r37451. |
|
| 37452 |
1 h 12 m |
amit |
/trunk/profitmandi-dao/src/main/java/com/spice/profitmandi/service/integrations/gstpro/ |
Fix e-invoice IRN failure when the NIC auth token is about to expire
NIC reissues the same auth token until it genuinely expires, so evicting the cache
inside the 5 minute leeway hands back the identical near-dead token. getAuthenticatedHeaders
checked the leeway once, before that re-fetch, and never inspected what came back.
NSUPHR1770 filed with a token that had 12 seconds of life left and the gateway answered
412/GSP752 'eInvoice AuthToken not found or expired'; the invoice filed a minute later,
past the expiry and so on a genuinely new token, succeeded.
getUsableAuthToken re-checks the refetched token and, when it is still inside a 60s
floor, waits out the remainder before minting again. The wait is bounded by that floor
and every caller on this path is a cron/async thread already running against a 60s NIC
timeout with no transaction held open.
Also make the gateway's own failures legible. NIC rejections arrive as
{Status, ErrorDetails}, but failures the ASP raises in front of NIC use
{status_cd, error{error_cd, message}}, which shares no field name with RespPl: Gson
produced an all-null object and the stored reason degraded to the literal
'RespPl{Status=0, Data=null, ErrorDetails=null, InfoDtls=null}', indistinguishable from
NIC rejecting the document. RespGSPErr already modelled that envelope but was a
non-static inner class and so could not be instantiated by Gson; made it static and
copy the code and message into ErrorDetails, which every caller already reads. |
|
| 37451 |
1 h 12 m |
amit |
/trunk/profitmandi-common/src/main/java/com/spice/profitmandi/common/services/ |
Send authenticated mail as the account that actually authenticates
googleMailSender logs in to smtp.gmail.com as sdtech@smartdukaan.com, but callers
build their From as noreply@smartdukaan.com. Google Workspace binds an authenticated
session to one identity and refuses the mismatch with 535 'Authorization failed:
Authenticated user is not authorized to send mail' - the password is accepted, the
message is not. Every alert routed through that sender was therefore lost at the SMTP
layer, the e-invoice failure alert for NSUPHR1770 among them.
AuthenticatedIdentityMailSender rewrites From to the sender's own username at send
time, keeping whatever display name the caller chose. Senders that do not authenticate
are left alone: gmailRelaySender is the IP-authorised relay and has no username, and
sendGridMailSender's username is the literal 'apikey', so both keep noreply@.
Enforced in the sender rather than at the ~40 call sites that build their own
MimeMessage and hardcode noreply@, so it covers the ones added next too. |
|
| 37450 |
1 h 20 m |
amit |
/trunk/profitmandi-cron/src/main/java/com/smartdukaan/cron/scheduled/ |
Oppo: find only the hole, and let a low-contrast hole be found at all
60% of oppo attempts never got past detection -- getMatCircles2 returned null and
the attempt was wasted before any aiming happened. That is a bigger loss than
everything the scheduling work addressed put together.
Two causes, both visible once real puzzles are captured and looked at.
1. param1 is the Canny HIGH threshold, and it was 100. Edges weaker than that are
discarded before circle finding begins. Measured hole-vs-background contrast on
four live puzzles: 98, 103, 28, 117. The 28 -- a pale lilac background -- cannot
produce an edge at 100, so no circle exists to find. Dropped to 50.
2. It insisted on TWO circles whose radii matched within 3px, and returned null
otherwise. So a puzzle where the hole was located perfectly still failed if the
ring was missed, or if the two radii differed by 4px.
The second is the sillier one: the ring is a DOM element,
dx_captcha_basic_sub-slider_, whose exact position aim() already reads via
pieceCentreX(). Detection was re-finding something known exactly, and then
throwing away a good hole because it could not confirm it. Now the ring position
is passed in and hough only has to find one thing.
Lowering param1 without the pairing check would let textured backgrounds (the
sand images especially) supply spurious circles with nothing to reject them, so
the candidate is verified as an actual hole: its core must be at least 15
luminance below the frame mean. Real holes measure 28-117 below, so 15 rejects
noise with margin.
Validated against live puzzles before committing:
ring is at rel 42px on every sample; hole lands at 113-194px, separation
71-152px -- so RING_EXCLUSION_PX = 30 never rejects a real hole
hole contrast 28-117 against MIN_HOLE_DARKNESS = 15
Not validated locally: the hough call itself. opencv 3.4.2-0 ships no osx/arm64
native and every JDK on this machine is arm64, so the detection path cannot run
here. param1 = 50 is reasoned from the measured contrasts, not measured. If the
detection rate does not move, 40 is the next value to try -- the darkness check is
what makes going lower safe.
Watch "Detected N circle(s) but all sat on the ring" and "Rejecting circle at Npx"
in the log: the first means hough found nothing but the ring, the second means the
darkness check is doing its job. |
|
| 37449 |
12 h 11 m |
amit |
/trunk/profitmandi-cron/src/main/java/com/smartdukaan/cron/scheduled/ |
Realme unmerged too: four selenium jobs, one per pool per brand
r37447 merged secondary and tertiary per brand to halve concurrent drivers.
r37448 reverted that for oppo, because serialising cost it more per imei than a
second browser did. This reverts it for realme as well, which ends the merge
experiment entirely.
What the merge actually cost realme is only visible now that oppo is parallel
again: realme's per-imei went 14.2s -> 29s. It did not change behaviour -- it is
simply competing with oppo's two browsers for CPU. At 29s its merged ceiling is
86400/29 = 2,979/day, just under the 3,054/day the daily re-check needs, so no
batch size could have closed the gap. Two parallel pools restore the ~3,900/day
it managed historically at four drivers.
Measured before this change (25 min window):
brand needed/day throughput note
Oppo 8,838 7,661 parallel revert worked, +122%
Realme 3,054 1,843 merged and CPU-starved
Vivo 10,082 16,128 surplus, cannot transfer to another brand
Sizes unchanged: oppo 25 per pool, realme 12 per pool, vivo 50+10.
Honest accounting of the merge: it was my idea, sized on per-brand arithmetic that
ignored contention between brands, and it is now fully reverted. What survives
from that line of work is the part that actually paid -- reaping orphaned browsers
(~1.8GB), the per-brand retry caps, and per-brand maxResults. Peak drivers are
back to 4, which is where they started.
Watch for contention: four browsers is the configuration that produced the 29s
figure for realme in the first place, so oppo may slow from its current 19.7s.
Re-measure both before tuning sizes again. |
|
| 37448 |
20 h 28 m |
amit |
/trunk/profitmandi-cron/src/main/java/com/smartdukaan/cron/scheduled/ |
Daily re-check for all brands; Oppo back to parallel pools
Two changes.
1. Re-check window 4 days (secondary) and 2 days (tertiary) -> 1 day everywhere.
Daily demand becomes the full universe rather than a fraction of it:
Oppo 4,798 + 4,040 = 8,838/day
Vivo 9,407 + 675 = 10,082/day (doing 15,345 -- fine)
Realme 1,973 + 1,081 = 3,054/day
2. Oppo's two pools run in PARALLEL again, reverting the merge in r37447 for that
brand only. Realme stays merged.
The merge was a straight trade of throughput for memory and oppo could not
afford it. Measured over 32 minutes and again over an hour the next morning:
3,555 then 3,456/day against 5,280 before merging. Batch cadence settled at a
very regular ~15.5 min per cycle, so a 30-imei merged batch takes ~10.5 min =
~21s/imei, against the 10.2s it managed unmerged. At 21s the ceiling is
86400/21 = 4,114/day even with zero idle, so no batch size and no shorter
fixedDelay could have reached 8,838. Serialising simply costs more per imei
here than running two browsers does.
Realme keeps the merge: it needs 3,054/day and delivers 2,952 merged, so a
small size bump covers it without a second browser.
Sizes: oppo 25 per pool (2 jobs in parallel), realme 12+12 merged, vivo 50+10
unchanged. Vivo has already cleared its entire secondary backlog -- the pool
reads 0 and both lists come back empty -- which is what the batch of 50 was for.
Cost: oppo goes back to two concurrent drivers, so the fleet is 3 rather than 2,
roughly +700MB. Acceptable against the ~2GB freed today by reaping orphaned
browsers and capping retries, but it is the reason realme was left merged.
Sizes are a starting point, not a final answer: oppo's per-imei time differs
markedly between merged and parallel modes, so re-measure before tuning further. |
|
| 37447 |
1 d 12 h |
amit |
/trunk/profitmandi-cron/src/main/java/com/smartdukaan/cron/scheduled/ |
Selenium: one browser per brand instead of one per pool
Oppo and Realme each ran secondary and tertiary as separate @Scheduled jobs, so
each brand opened two ChromeDrivers at once and the fleet ran four. Each driver
tree costs roughly 850MB. This box co-hosts a 9.4GB tomcat with ~3GB available
and has been OOM-killed twice this month -- tomcat was the victim both times, so
peak concurrency is the thing that matters here.
Combined into one job per brand. Nothing downstream changes: the two pools differ
only in which named query fills them, and both already funnel into the same
updateActivationDate -> checkWarranty -> saveActivation path. They are disjoint by
construction (secondary excludes anything with a FofoLineItem, tertiary is
FofoLineItem-based); distinct() is insurance, not a fix for a known overlap.
Sizing matters, because merging SERIALISES work that used to run in parallel and
keeping the old batch sizes would quietly cost throughput. Measured post-cap at
10.2s/imei (oppo, down from 14.6 after r37445) and 14.2s/imei (realme), solving
M * 86400 / (300 + M*t):
oppo 2 parallel jobs x10 = 4,299/day -> merged 15+15 = 4,277/day (parity)
realme 2 parallel jobs x10 = 3,910/day -> merged 10+10 = 2,959/day (-24%)
Oppo is sized to hold parity because it is already short of its 4,798/day need.
Realme is left at 20 -- it needs 2,243/day, so it can absorb the dip in exchange
for shorter batches and a shorter-lived browser.
What this saves and does not save: total driver-SECONDS are roughly unchanged,
which is the point of resizing. PEAK concurrent drivers halves from 4 to 2.
Also skips starting a browser at all when both pools come back empty -- currently
never true, but it costs nothing and a browser launched to do nothing is pure
waste on this box.
checkOppoImeiStatus/Tertiary and the realme equivalents are left in place for
manual invocation; they are simply no longer scheduled. |
|
| 37446 |
1 d 12 h |
amit |
/trunk/ |
Per-brand batch sizes, and stop chrome forking a GPU process it cannot use
maxResults was hardcoded in the shared repository methods, so Oppo and Vivo were
forced to the same secondary batch (10) and all three to the same tertiary (10).
It is now a parameter, set per brand at the call site.
Sizing is arithmetic, from measured IN-BATCH per-imei time. Solving
M * 86400 / (300 + M*t) = needed/day:
brand needed/day t M required set to
Vivo 9,407 0.8s 36 50 clears, ~12,700/day
Realme 1,973 13.4s 10 10 was 5 = ~1,177/day, short
Oppo 4,798 14.6s 88 10 HELD, see below
Correcting an earlier measurement of mine: I reported Vivo at 13.6s per imei and
concluded its backlog could not be cleared. That averaged across the ~300s idle
gaps BETWEEN batches. In-batch it is 0.8s -- Vivo is 17x faster than I said, is
idle ~97% of the time, and 50 clears its pool comfortably. There is no wait in
the Vivo path; it is simply fast.
Oppo is deliberately NOT raised. At 14.6s it would need M=88, which means
20-minute batches and near-permanent chrome sessions. But that 14.6s predates the
retry cap (r37445), which cuts exhausted imeis from 20 attempts to 7 and should
drop it sharply. Re-measure before sizing Oppo, rather than guessing high on a
box with 3GB free.
Also: --disable-gpu, --disable-dev-shm-usage, --disable-software-rasterizer on
both selenium tasks. Headless needs no GPU yet chrome forks a gpu-process per
browser -- 6 were alive across the fleet, pure overhead. No behaviour change.
Batch size does not raise peak concurrency (fixedDelay means one batch per job at
a time, so never more than 4 drivers). It raises DUTY CYCLE, which converts
chrome's footprint from intermittent to sustained. That matters here: tomcat is
9.4GB, available is ~3GB, and the two OOM kills this month both took tomcat.
Cron-only deploy. The dao signature change has no callers outside cron. |
|
| 37445 |
1 d 13 h |
amit |
/trunk/profitmandi-cron/src/main/java/com/smartdukaan/cron/scheduled/ |
Oppo/Realme: cap captcha retries per tick at 7 and 12, measured per brand
Most captcha work goes to imeis that never succeed. Measured over the last
window: oppo 6,245 attempts for 308 successes, realme 13,842 for 399. The bulk
is imeis grinding all 20 attempts in one already-refused session and getting
nothing -- oppo 279 of them, realme 610.
The imei is NOT abandoned. A captcha failure is a technical failure, not an
answer, so it stays pending and the next tick retries it in ~5 minutes with a
fresh page and session -- which the data suggests beats continuing in a session
the widget has refused seven times. This caps grinding, not retrying.
Per-brand caps, because the distributions genuinely differ:
OPPO cap 7 97.7% of successes kept, 58.4% of work saved (3,648 attempts)
Pre-glide, 33% of oppo's successes came from attempts 8-20 and 20 was the
right number. Post-glide (r37440) that is 2%: the drag lands first or second
try. cap 5 keeps 95.5%/saves 67.7%, cap 10 keeps 99.0%/saves 44.8%.
REALME cap 12 96.0% kept, 35.7% saved (4,937 attempts)
NOT 7. Realme has no glide, so its successes still spread to attempt 10+ and
a cap of 7 would cost it 16.5%. It also burns more than twice oppo's work, so
the gentler cap still saves more in absolute terms.
Combined: 8,585 of 20,087 attempts saved, ~43% less browser work, for ~3% fewer
successes per tick -- and those imeis come back next tick anyway. Less Chrome
work matters on this box: it co-hosts a 9.4GB tomcat, has 3.5GB free and has been
OOM-killed twice this month.
The two caps are independent constants and must not be synced. If glide is ported
to realme, expect its curve to shift left as oppo's did and its cap can drop to
~7 -- but measure it, do not assume it.
Windows are ~2h (oppo's only ~30 min post-glide), so the right numbers may drift.
Worth re-reading the attempt histogram after a full day. |
|
| 37444 |
1 d 15 h |
ranu |
/trunk/profitmandi-fofo/src/main/webapp/WEB-INF/views/ftl/ |
rbm drr dashboard new ui looks |
|
| 37443 |
1 d 15 h |
ranu |
/trunk/profitmandi-dao/src/main/java/com/spice/profitmandi/dao/model/ |
rbm drr dashboard new ui looks |
|
| 37442 |
1 d 15 h |
ranu |
/trunk/ |
rbm drr dashboard new ui looks |
|
| 37441 |
1 d 15 h |
ranu |
/trunk/profitmandi-cron/src/main/java/com/smartdukaan/cron/scheduled/ |
bi access to gaurav mathur |
|
| 37440 |
1 d 19 h |
amit |
/trunk/profitmandi-cron/src/main/java/com/smartdukaan/cron/scheduled/ |
Oppo: slide the drag in eased sub-moves instead of jumping
Aim is solved and it did not finish the job: r37438 lands the ring 0-1px from the
hole on 103 of 104 drags and oppo still converts ~15%, with POSITION_MISMATCH on
the rest. So the widget is refusing drags that end in the right place.
Dingxiang scores trajectory and velocity as well as final position and returns the
same POSITION_MISMATCH for both, so our verdict cannot separate "wrong place" from
"wrong-looking movement". The circumstantial case for the latter:
oppo 0.91s hold, 2-3 discrete jumps -> 15%
realme 1.93s hold, slower and more -> 54-74%
and r37438 made oppo FASTER by removing the per-step screenshots, moving it further
from realme's profile rather than closer.
glide() traverses each leg as up to 18 small sub-moves on an ease-out curve
(1-(1-t)^2, quick then settling) with a 45ms pause between, so the pointer
describes a continuous decelerating path instead of teleporting.
Nothing about the DISTANCE changes -- it is still exactly what the derived ratio
says, and verified to land on it precisely: a 116px leg becomes
[16,15,13,13,11,10,9,8,6,6,4,3,1,1] and sums to 116. Negative legs work the same
way for overshoot correction.
To be explicit, since this is the thing not to get wrong: the ratio remains fully
derived. It starts at zero and comes only from measured movement, per attempt and
per step. Splitting a leg into sub-moves gives MORE measured legs, not fewer, so
the estimate sharpens rather than being replaced by an assumption. No constant is
introduced -- MAX_GLIDE_STEPS, GLIDE_PX_PER_STEP and GLIDE_PAUSE_MS describe the
path, not the relationship between mouse px and ring px.
Expected hold ~1.7s against realme's 1.93s. Oppo only.
This is a hypothesis test, not a known fix. POSITION_MISMATCH rate answers it in
one tick; if it does not move, drag dynamics is eliminated and the remaining
suspect is Hough picking the wrong circle pair on oppo's imagery, which needs
visual evidence rather than another code change. |
|
| 37439 |
1 d 19 h |
amit |
/trunk/profitmandi-cron/src/main/java/com/smartdukaan/cron/scheduled/ |
Oppo: drop the stale nudgeFor javadoc that still referenced ASSUMED_RATIO
Comment only, no code change.
nudgeFor() and ASSUMED_RATIO were removed in r37438, but the javadoc describing
them was left orphaned above ALIGNED_PX, still explaining how to size a probe from
an assumed px-per-px. That is precisely the thing not to reintroduce, and a stale
comment arguing for it is an invitation to do so.
Replaced with a note on why there is no assumed ratio: it is derived from real
movement on every attempt and re-derived on every step within it. The constant was
tuned twice (1.7 -> 0.67 -> 1.7) against measurements contaminated by the leaked
slider press, and never helped. The widget's px-per-px genuinely varies by
environment (1.4-1.5 in production, 1.1-1.2 locally) and by brand, which is the
reason it has to be measured rather than assumed. |
|
| 37438 |
2 d 1 h |
amit |
/trunk/profitmandi-cron/src/main/java/com/smartdukaan/cron/scheduled/ |
Oppo: aim by tracking the ring element, not by re-detecting circles
r37437's closed loop could not close. Production, 13 minutes:
step 0 (probe) ok 36
lost the circles 32 (25 of them on step 1, right after the main move)
aligned within 2px 0
The probe worked and derived sane ratios (0.75-1.00), then the next measurement
returned null nearly every time and the loop bailed.
The cause is structural, not a tuning miss. getMatCircles2 needs Hough to resolve
TWO circles of similar radius, and as the ring closes on the hole they overlap and
stop being two. The measurement dies exactly at convergence, so no loop built on
"distance between two detected circles" can ever verify success. The old refine()
hit the same wall and silently released, which is why it never helped either.
So detection is now used ONCE, for what it is good at -- locating the hole while
the two are still far apart -- and from then on we track the ring's OWN element,
dx_captcha_basic_sub-slider_, whose rect stays readable however close it gets.
Validated on the live widget before committing:
bg element page x=556, width 288 CSS
ring centre page 598 -> 42 relative to bg
dark target 138 relative to bg -> page 694, gap 96px
The ring sits in the same 0-288 CSS space as OpenCV's x values, so bg.x + circleX
is directly comparable to the ring's position. That holds because getMatCircles2
crops by CSS coordinates and chrome runs --force-device-scale-factor=1. A 96px gap
matches the range production logs as "Distance is".
Also validated earlier that the ring's rect tracks a drag cleanly the whole way in
(+10 mouse -> +11 ring, repeatably), which is the property the circle detection
lacks.
The ratio is still derived per move and is now signed, so overshoot corrects itself
rather than needing to be detected. No screenshot or Hough pass per step either, so
the drag is quicker as well.
Oppo only. Realme is converting 78% and is not touched. |
|