| Rev |
Age |
Author |
Path |
Log message |
Diff |
| 37685 |
0 m |
amit |
/trunk/profitmandi-fofo/ |
Remove selenium from the fofo portal entirely
No browser starts in this WAR any more, and the selenium-java and
webdrivermanager dependencies are gone with it. Verified: no source reference to
selenium/WebDriver/ChromeDriver anywhere in the module, and zero selenium
artifacts on the runtime classpath.
Three pieces:
1. The insights schedule and its pull move to profitmandi-cron (r37684) and dao
(r37683). This service keeps only the READ paths -- redis, then in-memory,
then cs.agent_daily_insight -- plus refreshInsights(), which now asks the dao
sync service to run once and re-reads what it wrote. The portal therefore
holds no credentials at all: KNOWLARITY_USERNAME/PASSWORD are deleted from
this source, along with INSIGHTS_PAGE_URL and the 200-character CSS selector
the scrape depended on.
2. KnowlarityScraperService deleted. It was dead, not merely idle: zero
references anywhere, both @Scheduled annotations commented out, and its
@PostConstruct selenium block commented out with the note 'DISABLED - Live
status now comes from WebSocket via profitmandi-cron'. SVN backs that up --
r36057/r36058 (25-Mar) moved agent status to the websocket and r36072/r36075
(26-Mar) created KnowlarityBreakLogService in dao, but nobody removed the
corpse. It kept selenium in this WAR for six months after nothing used it.
The data agrees: 'On Break - <reason>' rows in cs.rbm_break_log stop on
25-27 March and plain 'Break' takes over, which is exactly that handover.
⚠ Consequence worth knowing: break-REASON granularity (lunch/meeting/sick)
was lost at that migration and is not coming back from the websocket feed.
3. setTokens() and POST /indent/set_knowlarity_tokens removed -- the method had
already been reduced to a log line, and the pull now authenticates itself per
run. Also retired the orphan knowlarity.scraper.enabled property (nothing read
it; it was still 'true' in prod), and corrected a stale section header and a
doc comment that promised 'current tokens' which no longer exist.
Not touched, deliberately: POST update_agent_status / bulk_update_agent_status /
update_status_by_name still exist and still write cs.rbm_break_log through
AgentLiveStatusService. They are orphaned -- the deleted scraper was their only
feeder and no view or script in the deployed WAR calls them -- but they are
public HTTP surface, so proving there is no INTERNAL caller is not the same as
proving no external one. Left for a separate decision. |
|
| 37684 |
0 m |
amit |
/trunk/profitmandi-cron/src/main/ |
Schedule the knowlarity insights pull here instead of in the fofo tomcat
Eight times a day, unchanged times (11:40, 13:40, 15:40, 17:40, 18:40, 19:15,
20:00, 20:40) so the shape of the day's data does not move. Calls
KnowlarityInsightsSyncService (dao r37683).
This lands next to KnowlarityCallMonitorScheduler on purpose: that one owns the
WebSocket status feed into cs.rbm_break_log, this one owns the periodic KPI pull
into cs.agent_daily_insight. They are the two halves of the same integration and
were previously split across two processes for no reason other than history.
What it replaces: the same schedule inside the fofo tomcat, where every run
started an ~850MB headless chrome on a box that holds a -Xmx8g tomcat and a
-Xmx2g cron jar on 16GB and has been kernel-OOM-killed twice. A run is now four
HTTPS calls, ~2-3 seconds.
⚠ Only fires under --spring.profiles.active=scheduled; a one-shot CLI run does
not start the schedulers.
staging.properties gains the knowlarity block. It had NO knowlarity keys at all,
which means the WebSocket call monitor has never run there either -- this fixes
both. Same credentials as prod; there is no separate SR tenant for staging.
Cadence is worth revisiting separately: the 8 slots were chosen when a run cost
75 seconds and 850MB. At 2 seconds, hourly or every 15 minutes during the
10:00-21:00 window (matching the call monitor) would be nearly free. |
|
| 37683 |
1 m |
amit |
/trunk/profitmandi-dao/src/main/java/com/spice/profitmandi/service/knowlarity/ |
Move the knowlarity insights pull into dao, and delete the dead earlier attempt
The pull belongs in dao because its SCHEDULE moves to profitmandi-cron while the
fofo portal still reads the same rows. Both modules depend on dao, so that is the
only place the two can share it.
KnowlarityInsightsSyncService fetches via KnowlarityApiClient (moved here from
fofo, unchanged) and upserts cs.agent_daily_insight. No DDL: every column it
writes already exists.
It writes what the old chrome scrape could not:
- the *_seconds columns. Verified on prod over 7 consecutive days, every row had
logged_in_seconds, break_seconds, available_seconds and talk_seconds = 0. The
API returns fractional seconds as numbers, so they are stored directly rather
than re-parsed out of a display string.
- calls_answered / missed_calls / total_calls, also 0 on every row, because the
scrape never captured them at all.
auth_id resolution keeps the email-PREFIX match from the fofo version: some
stored emails are truncated ('user@smartdukaan.co..'), so matching the whole
address silently misses those agents.
Redis keys are deliberately the same pair the portal reads, so its fast path
stays warm across the process boundary. ⚠ That makes AgentInsightRow's getter
names a contract with fofo's AgentInsightDTO -- rename one without the other and
the portal deserialises blanks.
Also deleted, all with zero external references:
- KnowlarityAgentInsightsService: an earlier, unfinished run at this same API.
Its ALT_INSIGHTS_APIS list of guessed endpoints is the giveaway that it never
worked; nothing ever called it. It owned three redis keys nobody read
(knowlarity:agent:insights, :summary, :lastUpdate) -- distinct from the live
knowlarity:insights:* pair, which stays.
- AgentInsightsDTO / AgentInsightsSummaryDTO: used only by that service.
- KnowlaritySocketTest: a scratch main(), never referenced. |
|
| 37682 |
1 h 13 m |
amit |
/trunk/profitmandi-dao/src/main/java/com/spice/profitmandi/service/warehouse/ |
feat(warehouse): auto-approve backdated POs in the creator's name
A backdated PO used to be parked in POStatus.INIT until an HOD clicked a mailed approval link. INIT
keeps a PO out of the PO list, out of GRN/invoice matching and out of the auto-close sweep, so an
unclicked PO was frozen rather than merely pending - 440 had accumulated since 2024, of 2547 approval
rows only 2023 were ever clicked.
- every PO now goes READY at creation, backdated or not
- the warehouse.poapproval row is still written, stamped approvedOn=now and approvedBy=<creator> by
recordAutoApproval(), since that row is what identifies a PO as backdated and who backdated it
- the HOD mail survives as a notification: names the creator, no approval link, PDF still attached
- /approve-po-date/* endpoints kept so links already sent do not 404
Completes r37679-81. Backfill for the three POs still inside POScheduler's window is staged at
profitmandi-dao/src/main/resources/sql/backdated_po_auto_approval_backfill_20260917.sql, not run. |
|
| 37681 |
1 h 24 m |
amit |
/trunk/profitmandi-web/src/main/java/com/spice/profitmandi/web/v2/controller/fofo/ |
feat(warehouse): resolve the logged-in user on V2 PO creation and GRN mismatch resolution
V2 mirror of r37680, keeping the dormant V2 controllers in step with the fofo MVC ones.
- createPurchaseOrder resolves the logged-in user via getEmailId + authRepository and sets createdBy
- resolvedMismatchRequest passes the resolver through
Pairs with r37679. |
|
| 37680 |
1 h 25 m |
amit |
/trunk/profitmandi-fofo/src/main/ |
feat(warehouse): resolve the logged-in user on PO creation and GRN mismatch resolution
Backdated POs are auto-approved in their creator's name, so the user has to travel with the request.
- createPurchaseOrder resolves the logged-in user via cookiesProcessor + authRepository and sets createdBy
- resolvedMismatchRequest passes the resolver through, for the correction PO it can raise
- PO create page no longer says backdated POs need HOD approval
Pairs with r37679. |
|
| 37679 |
1 h 25 m |
amit |
/trunk/profitmandi-dao/src/main/ |
feat(warehouse): carry the PO creator through purchase order creation for backdated auto-approval
Backdated POs are to be auto-approved in the name of whoever raised them, but the service had no way
to know who that was: the model carried no user and buyerId is the seller, not a person.
- CreateWarehousePurchaseOrderModel carries createdBy
- resolvedMismatchRequest takes resolvedBy, so the correction PO raised on a GRN price mismatch -
dated to the supplier invoice and therefore backdated by nature - is credited to whoever resolved it
- staged backfill SQL for the three backdated POs still stuck in INIT inside POScheduler's
auto-close window; data only, no schema change, not yet run
The release-to-READY change in PurchaseOrderServiceImpl follows separately. |
|
| 37678 |
1 h 58 m |
amit |
/trunk/profitmandi-fofo/src/main/java/com/spice/profitmandi/web/service/ |
Replace the knowlarity insights chrome scrape with SR's own JSON API
Drops the last unattended headless chrome out of the fofo tomcat. The scheduled
insights job ran 8 times a day and each run started an ~850MB chrome tree inside
the tomcat JVM's host -- a box holding -Xmx8g tomcat plus a -Xmx2g cron jar on
16GB that has been kernel-OOM-killed twice with tomcat the victim.
It was also losing data the whole time. Verified on prod across 7 consecutive
days: every row of cs.agent_daily_insight has logged_in_seconds, break_seconds,
available_seconds, talk_seconds, calls_answered, missed_calls and total_calls
set to 0. Two separate causes, both fixed here:
- parseTimeToSeconds split on ':' expecting 'HH:mm:ss', but the table renders
'2h 50m 39s'. A one-element split fell through to return 0, and because it
never reached the NumberFormatException branch it did not even warn. It now
parses the h/m/s shape and still accepts HH:mm:ss and HH:mm.
- the scrape never captured the call counts at all. The API carries them.
The auth chain is four calls and is not guessable, so it is documented in
KnowlarityApiClient: POST /vr/sr_login/ establishes the session and returns an
HS256 token that the API REJECTS; GET /newsr/user_details yields new_sr_ui_url
carrying a one-shot SSO blob (in a browser this hop is javascript, so it is
invisible to anything that merely follows redirects); POST /vr/sso_login/
exchanges that blob for the RS256 token the API accepts, whose claims embed the
srsessionid and so bind it to the session; GET /newsr/agents_insights/ with
header jwtAuthorization. Wrong token and wrong header name both answer
'Invalid token', so the error never tells you which mistake you made. The window
parameters are start_time/end_time -- start_date/end_date authenticates fine and
returns 'Error in API'.
Redirects are followed BY HAND. setInstanceFollowRedirects(true) exposes only
the final response's headers, so the Set-Cookie issued on the intermediate hops
is lost, the session never forms and user_details answers with an HTML error
page. This cost a debugging cycle; the reason is commented at the call site.
Verified against the live account before committing: 13 agents returned,
calls_offered == calls_answered + missed_calls holds for all 13, durations match
the previous scrape to within the elapsed window (~55s), and formatSeconds /
parseTimeToSeconds round-trip cleanly over the real values.
DTO fields stay the same display strings ('2h 50m 39s'), so every existing
reader is unaffected; only the previously-zero numeric columns change.
KnowlarityScraperService still uses ChromeDriver for operator-triggered break-log
scrapes, so the selenium dependency stays for now. Its @Scheduled annotations are
already commented out, so nothing unattended starts a browser any more. |
|
| 37677 |
2 h 27 m |
ranu |
/trunk/profitmandi-cron/src/main/java/com/smartdukaan/cron/scheduled/b2b/ |
price drop and hike fixed |
|
| 37676 |
2 h 34 m |
ranu |
/trunk/profitmandi-web/src/main/java/com/spice/profitmandi/web/controller/ |
v2 version some fixes |
|
| 37675 |
3 h 2 m |
amit |
/trunk/profitmandi-cron/src/main/java/com/smartdukaan/cron/ |
Remove the dead Samsung and Amazon Selenium paths
Both were manual one-shots that nothing runs any more, and each started an
850MB chrome tree that now has to queue on the browser lane, so they are cost
without a caller.
Evidence they are dead rather than merely idle:
- No crontab entry, no cron.d file and no script references --samsung or
--amazonPurchase, and neither flag appears even once in cron.log.
- fofo.activated_imei has ZERO Samsung rows written by the cron (auth_id 0) in
the last 90 days. All 2,331 Samsung rows in that window are auth_id 307, i.e.
the human CSV upload, most recently 16-Sep. The scraper is not what keeps
Samsung current; people are.
- ScheduledSkeleton.fetchImeiActivation() had already been retired in place --
its @Scheduled was commented out with 'No longer scheduled'.
- RunOnceTasks.amazonPurchase() reads /Users/amit/Downloads/amazon.xlsx, a
laptop path that cannot exist on the server.
Removed: SamsungIMEIActivationService, the whole scheduled/amazon package
(AmazonPurchaseService, OrderSummary, OrderRow, AmazonUser), their RunOnceTasks
callers and helpers (fetchImeiActivation, amazonPurchase, getOrderSummary,
parseRow), the two Application CLI blocks and the retired ScheduledSkeleton
wrapper. The amazon package had no importers outside RunOnceTasks.
Both also leaked a chrome profile dir on every run -- AmazonPurchaseService
never called quit() at all, and SamsungIMEIActivationService called it outside
any finally -- so this removes two leak sources rather than fixing them.
Untouched: RunOnceTasks.mailDashboardScreenshots() is a third dead Selenium
one-shot (also zero invocations) but it mails a report, so it is left for a
separate decision. |
|
| 37674 |
4 h 41 m |
amit |
/trunk/profitmandi-fofo/src/main/java/com/spice/profitmandi/web/service/ |
Hold the browser lane around the knowlarity scrapes
These run Chrome inside the tomcat JVM, on the same box as the cron jar's
oppo/realme imei lane, with nothing coordinating the two. isScraping only guards
this JVM and cannot see the cron's driver, so the interlock has to be the
OS-level one -- BrowserLane, common r37672.
Fifteen minutes rather than the cron side's five: the insights scrape runs only
8 times a day against a lane that is busy ~96% of the time, so a short timeout
here would mean the insights never refreshed. On contention it serves cache.
The break-log scraper takes the lane around its freshDriver, which has a clean
try/finally bracket. setupDriver()'s long-lived field driver is deliberately left
alone -- its lifetime spans init to @PreDestroy and holding an OS lock that long
would starve the imei lane. |
|
| 37673 |
4 h 41 m |
amit |
/trunk/profitmandi-cron/src/main/java/com/smartdukaan/cron/scheduled/ |
Hold the browser lane around the oppo/realme/motorola drivers
Takes BrowserLane (common r37672) before new ChromeDriver and releases it after
quit() returns -- the 850MB is resident for the whole chunk, not just at
startup, so bracketing only the constructor would protect nothing.
Five minutes of waiting, then give up and return what we have: the only things
that can hold the lane that long are the other brand mid-chunk or the knowlarity
scrape in the fofo tomcat, and giving up costs nothing because an unstamped imei
stays pending and the lane's next turn picks it up.
CheckMotorolaWarrantyTask carries a comment saying 'do not let this job overlap
the oppo/realme window' that nothing ever enforced. It is wired here too so it is
already safe whenever it gets a trigger -- it still has none today.
Note on sizing, since the obvious knob is the wrong one: shrinking CHUNK from 25
was evaluated and rejected. The idle window is a fixed 20s bolted onto a variable
work period, so 25->10 moves the duty cycle only 96% -> 91% while costing 8.4%
of daily throughput and 2.5x the driver launches. The lever for duty cycle is the
fixedDelay gap, not the chunk size. |
|
| 37672 |
4 h 41 m |
amit |
/trunk/profitmandi-common/src/main/java/com/spice/profitmandi/common/util/ |
Add BrowserLane: machine-wide interlock so only one headless Chrome runs at a time
Every ChromeDriver tree on the app box costs ~850MB RSS, on a host holding a
-Xmx8g tomcat and a -Xmx2g cron jar on 16GB that has been kernel-OOM-killed
twice with tomcat the victim. Peak concurrent drivers is the number that
matters, not drivers per day.
Measured on the oppo/realme lane: a 25-imei chunk takes 505s (12s driver
startup, then ~19.7s an imei) against a fixed 20s fixedDelay, i.e. a 96% duty
cycle. Meanwhile tomcat starts its own driver 8 times a day for the knowlarity
scrape. On 17-Sep that scrape ran 11:40:00-11:41:07 and the lane's next driver
came up at 11:42:21 -- it cleared by 74 seconds, by luck. At a 96% duty cycle
the collision cannot be dodged by choosing better times, only by serialising.
The two JVMs are separate processes owned by different users, so an in-process
flag cannot see across the boundary. Uses an OS file lock at
/var/lock/sd-browser.lock (overridable via -Dsd.browser.lock), created 0666 on
first use because cron runs as root and tomcat as tomcat and whichever starts
first creates it. A file lock rather than a marker file so the kernel reclaims
it when the holder dies: verified by SIGKILLing the holder mid-hold, the next
JVM acquired in 264ms. A ReentrantLock gates it first, since a second thread in
the same JVM would otherwise get OverlappingFileLockException rather than block.
Callers wait rather than skip. The knowlarity scrape needs 67s against a lane
busy 96% of the time, so skip-on-contention would mean it essentially never
refreshed again. Eight scrapes a day costs the imei lane about nine minutes of
waiting in total.
Fails open if the lock file is unusable (missing /var/lock, read-only mount, dev
laptop): serialising within the JVM only, rather than stopping imei activation
over what is a safety interlock. |
|
| 37671 |
6 h 20 m |
ranu |
/trunk/ |
loi revival process modify |
|
| 37670 |
21 h 24 m |
amit |
/trunk/profitmandi-fofo/src/main/ |
Check GSTIN shape on the onboarding screen before submit (jsVersion 425)
Both GST boxes are validated client-side and must match, so ops see the problem on the screen
instead of discovering it when NIC refuses the first invoice. Server-side guard in dao r37669
is what actually enforces it. |
|
| 37669 |
21 h 24 m |
amit |
/trunk/profitmandi-dao/src/main/java/com/spice/profitmandi/service/user/ |
Reject an invalid or mismatched partner GSTIN before it is saved
The onboarding screen carries the GSTIN in two boxes both labelled GST NUMBER - the retailer
one (dtr.retailer.number) and the store one (fofo_store.gst_number, which billing reads).
Neither was validated, so DLSH1660 was saved with the LOI's pincode in the store box while the
retailer box held the real GSTIN, and its first invoice would have been refused by NIC.
validateGstNumbers runs before any write (covers the fofo portal, the V2 API and the partner
app, which all funnel through updateRetailerDetails): both values are normalised, both must be
valid GSTINs, and when both are present they must agree - across 1,800 partners on record they
already do, bar the one that caused this. Needs common r37668. |
|
| 37668 |
21 h 24 m |
amit |
/trunk/profitmandi-common/src/main/ |
Validate GSTIN shape, state code and checksum
isValidGstNumber accepted any 15-character string, so a pincode (110095, rejected only for
length) and junk (Hdjiekwbdbsjskz) reached fofo_store.gst_number, which billing sends to NIC
as the recipient - the invoice is then refused and the goods cannot ship.
Now: 2-digit state code + PAN + entity + Z + mod-36 check digit, trimmed and upper-cased.
Blank still passes (the field is optional); callers needing one present check that themselves.
Checked against live data: 1,569 of 1,570 well-formed partner GSTINs pass, the one that fails
(HRSP330) has a genuinely wrong check digit.
Adds RTLR_1020 / RTLR_1021 for the invalid and mismatched cases. |
|
| 37667 |
22 h 32 m |
amit |
/trunk/profitmandi-dao/src/main/resources/sql/ |
chore(sql): drop vendor catalog pricing for supplier 1 Hotspot, flagged internal 2026-09-16 (as run on prod; backups _bak_vcp/_vcpl_internal_hotspot_20260916) |
|
| 37666 |
22 h 57 m |
amit |
/trunk/profitmandi-web/src/main/java/com/spice/profitmandi/web/controller/ |
Hot Deal brand: /fofo/hotDeals/brands accepts categoryId, so the brand chips match the active tab |
|