| Rev |
Age |
Author |
Path |
Log message |
Diff |
| 37499 |
4 d 11 h |
ranu |
/trunk/ |
sales target and achievement added cron and flags |
|
| 37482 |
5 d 9 h |
amit |
/trunk/profitmandi-cron/src/main/java/com/smartdukaan/cron/scheduled/ |
carlcare: recover tecno/itel activation dates, and fix trunk broken by r37479
r37479/r37480 swept a working-copy edit to Application.java into the mail commit: trunk
has called scheduledTasks.checkCarlcareImeiActivation() since then without containing it,
so profitmandi-cron has not compiled. This adds the rest.
WHAT THIS IS
The tecno feed has been dead since 2024-11-20, when transsion decommissioned the SAP
OData hosts cms.tecno-mobile.com:8099 / cms.itel-mobile.com:8099. The imwav DCR portal
recovers itel but has nothing for tecno -- all three tecno logins together expose 17
imeis over five years, against 31,928 tecno rows in fofo.activated_imei. Everything since
has been manual CSV via /imei/upload.
Carlcare is transsion's own after-sales arm, and the warranty-check page on carlcare.in is
backed by a public per-imei endpoint that answers for tecno, itel and infinix with no
login, no cookie and no captcha:
GET ind-mis-carl.shalltry.com/CarlcareClient/electronic-card/check-extended_warranty-web?imei=
sign: md5(SALT + imei)
The sign header is mandatory (without it: code 10022 "Sorry, web sign is error"). The salt
is in the site's own bundle, _nuxt/085537c.js module 688, along with the base url; re-read
that bundle if it ever stops working. status 3 = activated and carries activeTime, status
2 = device known but not activated yet and activeTime is null.
It is also the semantically right source. The DCR portal serves an INVENTORY report and
the old SAP feed served a TERTIARY SALES report, whereas activeTime is the date the
handset was actually activated -- which is what fofo.activated_imei is meant to hold and
what tertiary payout is computed on.
Measured against hadb1 before writing any of this: tecno 355463920708766 -> 2026-08-28 and
itel 359207322028000 -> 2026-08-30, both exact matches to rows we already had. A read-only
trial of 50 itel imeis answered 50/50 with no errors and no sign rejections, agreed with
our catalog brand on all 50, and returned a date for 18 -- every one of those 18 falling
AFTER our own billing date, 3 to 1,111 days, median ~220.
SHAPE
Pool queries and saveActivation semantics are the vivo ones, so the two read alike, and no
DAO change was needed: the pending queries are already brand-generic. The far end is far
cheaper than vivo's, one signed GET per imei, so there is no captcha service, no cookie
store, no session seeding and no verdict reporting.
50 imeis every 5 minutes = 14,400 lookups a day against ~10,400 pending itel, so the whole
pool is covered daily with headroom. The pool reaches back to 2021, so the ticks are
themselves the backfill of the nov-2024 blackout; there is no one-off to run.
Two things worth knowing before changing it:
- The batch is drawn ROUND-ROBIN across each (brand, channel) queue, not by concatenating
them. A full daily pass can concatenate freely because it walks to the end, but a
50-at-a-time tick cannot: itel is ~1,550 secondary against ~8,851 tertiary, so the head
of a concatenated list is ~31 straight ticks of pure secondary before one tertiary imei
is asked about. The first trial batch was 100% secondary for exactly that reason.
- Every outcome stamps the row, failures included. This is the one deliberate departure
from vivo, which leaves a failure unrecorded so it retries next pass -- safe there
because the next pass is tomorrow. On a 5-minute cadence it is not: an unstamped imei is
due again in five minutes, the query keeps handing back the same 50 rows, the batch never
advances past them, and the endpoint is asked the same questions twelve times an hour for
as long as it keeps failing. See the runaway documented on oppoRealmeImeiActivation.
BRANDS is itel alone to start. That is a rollout order, not a limit of the endpoint: itel
is the brand we can CHECK, because the DCR pull independently produces itel dates to
reconcile against, where tecno has nothing. Add "Tecno" once a day's rows agree. Both feeds
may write itel meanwhile with no coordination -- the pool query only returns imeis whose
activationTimestamp is still null, so whatever one fills has left the other's pool.
Scheduled tick plus a --checkCarlcareImeiActivation flag, both wired. |
|
| 37472 |
5 d 11 h |
amit |
/trunk/profitmandi-cron/src/main/java/com/smartdukaan/cron/ |
IMEI activation: one snapshotted daily pass per brand, on one thread, with per-brand metrics
Four @Scheduled jobs every 5 minutes become two daily passes. Oppo and realme
share one thread and alternate in 25-imei chunks, so exactly one ChromeDriver is
alive at a time instead of four; vivo keeps its own thread since it is direct
HTTP and does not contend for a browser.
The pass snapshots its pool before any browser starts and walks that list to the
end. It never re-queries, and that is the actual fix. A failed lookup never
reaches dateMap.put, so no row is written, so createTimestamp is not bumped, so
the imei was eligible again on the next tick five minutes later. Measured 29-Aug:
realme issued 4,524 requests against 1,004 distinct imeis -- 4.5 asks each, 78%
of the day's budget spent re-asking -- while oppo, which rarely fails, sat at
1.03. More requests hardened the block, which caused more failures. A pass bounds
that: a failure costs one retry tomorrow, never one in five minutes.
This supersedes the r37447/r37448/r37449 argument about driver count, which was
about the wrong variable. That argument blamed realme's collapse on CPU
contention pushing the captcha render past the element waits. The logs do not
support it: on 29-Aug oppo took ZERO canvas timeouts across all 24 hours on the
same box, same six cores, same driver count, same captcha vendor, load average
0.9 -- including the 15:00-23:00 window in which realme solved nothing at all.
Realme's own canvas wait is 15s against oppo's 8s, so the longer wait is the one
expiring. What realme's timeout rate tracks is its own daily request volume, and
it resets at midnight: 920/day -> 0.3%, 3,467/day -> 28%, 4,524/day -> 75%. That
is realme.com declining to serve the widget.
DAYS=0 is deliberate and is not an off-by-one: the pool filter is
createTimestamp < now().atStartOfDay().minusDays(DAYS), so DAYS=1 measures
against yesterday midnight and silently yields a two-day cadence, which is what
oppo and realme were running.
Sizing measured on prod for a midnight start: oppo 4,133 and realme 2,118 imeis,
11.7h + 8.4h = 20.1 hours of a single thread. It fits with no slack; if the
'pass finished' counts come in short of 'pass starting', the lever is DAYS=1
rather than a second thread.
Observability: ImeiActivationGauges publishes the funnel per brand on
/actuator/prometheus, which alloy already scrapes on this host -- due, churned,
captcha_shown, captcha_solved, answered, dates_found, errors, run_seconds and
last_finish_epoch. Each stage fails differently and says what broke. Rates are
left to PromQL. The stage that matters for health is answered: churned>0 with
answered==0 is precisely the shape of both silent outages this year (oppo wrote
nothing for a week; the vivo captcha solver was dead for 46 days). dates_found is
deliberately NOT a health signal -- when the multi-year backlog drained at the
end of August, yield fell from ~100% to 2-3% on the same day across all three
brands with nothing broken.
Nagios cleanup: the Nagios server and every NRPE daemon are gone, so
WriteToPropertiesFile and the commented-out blocks that fed
nagios-cron.properties are deleted, and NagiosMonitorTasks is renamed
BalanceMonitorTasks for the transport it actually uses. Noted there that nothing
calls it -- there is no @Scheduled entry and no other caller -- which is why both
balance gauges have always read -1. |
|
| 37449 |
8 d 6 h |
amit |
/trunk/profitmandi-cron/src/main/java/com/smartdukaan/cron/scheduled/ |
Realme unmerged too: four selenium jobs, one per pool per brand
r37447 merged secondary and tertiary per brand to halve concurrent drivers.
r37448 reverted that for oppo, because serialising cost it more per imei than a
second browser did. This reverts it for realme as well, which ends the merge
experiment entirely.
What the merge actually cost realme is only visible now that oppo is parallel
again: realme's per-imei went 14.2s -> 29s. It did not change behaviour -- it is
simply competing with oppo's two browsers for CPU. At 29s its merged ceiling is
86400/29 = 2,979/day, just under the 3,054/day the daily re-check needs, so no
batch size could have closed the gap. Two parallel pools restore the ~3,900/day
it managed historically at four drivers.
Measured before this change (25 min window):
brand needed/day throughput note
Oppo 8,838 7,661 parallel revert worked, +122%
Realme 3,054 1,843 merged and CPU-starved
Vivo 10,082 16,128 surplus, cannot transfer to another brand
Sizes unchanged: oppo 25 per pool, realme 12 per pool, vivo 50+10.
Honest accounting of the merge: it was my idea, sized on per-brand arithmetic that
ignored contention between brands, and it is now fully reverted. What survives
from that line of work is the part that actually paid -- reaping orphaned browsers
(~1.8GB), the per-brand retry caps, and per-brand maxResults. Peak drivers are
back to 4, which is where they started.
Watch for contention: four browsers is the configuration that produced the 29s
figure for realme in the first place, so oppo may slow from its current 19.7s.
Re-measure both before tuning sizes again. |
|
| 37448 |
8 d 15 h |
amit |
/trunk/profitmandi-cron/src/main/java/com/smartdukaan/cron/scheduled/ |
Daily re-check for all brands; Oppo back to parallel pools
Two changes.
1. Re-check window 4 days (secondary) and 2 days (tertiary) -> 1 day everywhere.
Daily demand becomes the full universe rather than a fraction of it:
Oppo 4,798 + 4,040 = 8,838/day
Vivo 9,407 + 675 = 10,082/day (doing 15,345 -- fine)
Realme 1,973 + 1,081 = 3,054/day
2. Oppo's two pools run in PARALLEL again, reverting the merge in r37447 for that
brand only. Realme stays merged.
The merge was a straight trade of throughput for memory and oppo could not
afford it. Measured over 32 minutes and again over an hour the next morning:
3,555 then 3,456/day against 5,280 before merging. Batch cadence settled at a
very regular ~15.5 min per cycle, so a 30-imei merged batch takes ~10.5 min =
~21s/imei, against the 10.2s it managed unmerged. At 21s the ceiling is
86400/21 = 4,114/day even with zero idle, so no batch size and no shorter
fixedDelay could have reached 8,838. Serialising simply costs more per imei
here than running two browsers does.
Realme keeps the merge: it needs 3,054/day and delivers 2,952 merged, so a
small size bump covers it without a second browser.
Sizes: oppo 25 per pool (2 jobs in parallel), realme 12+12 merged, vivo 50+10
unchanged. Vivo has already cleared its entire secondary backlog -- the pool
reads 0 and both lists come back empty -- which is what the batch of 50 was for.
Cost: oppo goes back to two concurrent drivers, so the fleet is 3 rather than 2,
roughly +700MB. Acceptable against the ~2GB freed today by reaping orphaned
browsers and capping retries, but it is the reason realme was left merged.
Sizes are a starting point, not a final answer: oppo's per-imei time differs
markedly between merged and parallel modes, so re-measure before tuning further. |
|
| 37447 |
9 d 6 h |
amit |
/trunk/profitmandi-cron/src/main/java/com/smartdukaan/cron/scheduled/ |
Selenium: one browser per brand instead of one per pool
Oppo and Realme each ran secondary and tertiary as separate @Scheduled jobs, so
each brand opened two ChromeDrivers at once and the fleet ran four. Each driver
tree costs roughly 850MB. This box co-hosts a 9.4GB tomcat with ~3GB available
and has been OOM-killed twice this month -- tomcat was the victim both times, so
peak concurrency is the thing that matters here.
Combined into one job per brand. Nothing downstream changes: the two pools differ
only in which named query fills them, and both already funnel into the same
updateActivationDate -> checkWarranty -> saveActivation path. They are disjoint by
construction (secondary excludes anything with a FofoLineItem, tertiary is
FofoLineItem-based); distinct() is insurance, not a fix for a known overlap.
Sizing matters, because merging SERIALISES work that used to run in parallel and
keeping the old batch sizes would quietly cost throughput. Measured post-cap at
10.2s/imei (oppo, down from 14.6 after r37445) and 14.2s/imei (realme), solving
M * 86400 / (300 + M*t):
oppo 2 parallel jobs x10 = 4,299/day -> merged 15+15 = 4,277/day (parity)
realme 2 parallel jobs x10 = 3,910/day -> merged 10+10 = 2,959/day (-24%)
Oppo is sized to hold parity because it is already short of its 4,798/day need.
Realme is left at 20 -- it needs 2,243/day, so it can absorb the dip in exchange
for shorter batches and a shorter-lived browser.
What this saves and does not save: total driver-SECONDS are roughly unchanged,
which is the point of resizing. PEAK concurrent drivers halves from 4 to 2.
Also skips starting a browser at all when both pools come back empty -- currently
never true, but it costs nothing and a browser launched to do nothing is pure
waste on this box.
checkOppoImeiStatus/Tertiary and the realme equivalents are left in place for
manual invocation; they are simply no longer scheduled. |
|
| 37395 |
11 d 6 h |
amit |
/trunk/profitmandi-cron/src/main/java/com/smartdukaan/cron/ |
observability: retire the dead Nagios monitoring path, migrate balance signals to Micrometer
There is no Nagios server and the NRPE daemons have been removed from every
host, so the properties files written under /var/log/services were being
produced for nobody.
Deleted:
- CronMetricsService: zero references. Spring instantiated the @Service so its
constructor registered three meters, but nothing ever incremented them --
cron_execution_* read 0 in prometheus while cron_job_count_total (from
CronJobMonitorAspect) had recorded 1,597 executions.
- NagiosMonitoringScheduledSkeleton.
Removed the nagios properties writes from ScheduledSkeleton (-155 lines) and
KnowlarityCallMonitorScheduler (-46, including its now-purposeless
markTaskComplete helper). These recorded a per-job 0/1 status that
CronJobMonitorAspect already captures properly as success, failure and
duration metrics.
Kept the two signals that are genuinely useful: SMS gateway balance and the
ThinkWalnut recharge wallet balance now publish as Micrometer gauges via the
new BalanceGauges component, scraped from /actuator/prometheus. -1 means 'not
read yet' so a scrape before the first run is distinguishable from a real zero.
No behaviour change to any scheduled job. Compiles clean. |
|
| 37292 |
25 d 11 h |
amit |
/trunk/profitmandi-cron/src/main/java/com/smartdukaan/cron/scheduled/ |
Hot deals: javadoc matches reindex-based sync
syncHotDealFlags no longer mirrors model_hot_deal into tag_listing.hot_deals;
since r37276 hot_deal_b is derived from the active window at index time, so the
daily job re-indexes models whose window opened today or closed yesterday.
Comment-only, no behaviour change. |
|
| 37258 |
30 d 10 h |
amit |
/trunk/profitmandi-cron/src/main/java/com/smartdukaan/cron/ |
Schedule syncHotDealFlags daily 00:20 (ScheduledSkeleton) + --syncHotDealFlags CLI arg: completes r37255 - without the skeleton trigger the tag_listing.hot_deals mirror never runs, so deal windows would not start/expire in Solr |
|
| 37234 |
32 d 7 h |
vikas |
/trunk/profitmandi-cron/src/main/java/com/smartdukaan/cron/scheduled/ |
Updated Gmail Timeout |
|
| 37230 |
32 d 8 h |
vikas |
/trunk/profitmandi-cron/src/main/java/com/smartdukaan/cron/scheduled/ |
Updated Gmail Timeout |
|
| 37172 |
37 d 14 h |
amit |
/trunk/profitmandi-cron/src/main/java/com/smartdukaan/cron/ |
PJP agenda instances: nightly 23:45 sync cron + manual run arg |
|
| 37162 |
37 d 16 h |
vikas |
/trunk/profitmandi-cron/src/main/java/com/smartdukaan/cron/scheduled/ |
Shopify Inventory Sync |
|
| 37159 |
38 d 10 h |
amit |
/trunk/profitmandi-cron/src/main/java/com/smartdukaan/cron/ |
Nightly partner identity integrity monitor (7 violation classes over fofo_store/user.user/user_accounts/user_role incl. same-GST/mobile multi-open-store mandate checks); scheduled 07:00 + CLI trigger partnerIdentityIntegrityMonitor |
|
| 37083 |
53 d 13 h |
amit |
/trunk/profitmandi-cron/src/main/java/com/smartdukaan/cron/scheduled/ |
Wire noon price-drop cron to per-drop batch (PriceDropBatchService)
@Scheduled("0 0 12") now delegates to
BatchScheduledTasks.reprocessPriceDropsWithBatch() -> PriceDropBatchService,
so each price drop commits in its own REQUIRES_NEW transaction instead of one
~20-min transaction holding user_wallet locks and timing out live
wallet/order/scan traffic at noon.
Requires profitmandi-dao PriceDropBatchService / PriceDropProcessingHelper. |
|
| 37040 |
58 d 13 h |
ranu |
/trunk/profitmandi-cron/src/main/java/com/smartdukaan/cron/ |
on billing delay partner map to focus and revival |
|
| 36993 |
65 d 13 h |
amit |
/trunk/profitmandi-cron/src/main/java/com/smartdukaan/cron/ |
Thread dryRun through sendCreditNote; --sendCreditNote defaults to dry-run (--live for real); add 06:00-on-1st monthlyMarginsCnDryRun scheduled job |
|
| 36552 |
113 d 7 h |
ranu |
/trunk/profitmandi-cron/src/main/java/com/smartdukaan/cron/scheduled/ |
changeList |
|
| 36511 |
118 d 10 h |
amit |
/trunk/profitmandi-cron/src/main/java/com/smartdukaan/cron/ |
Add daily inventory snapshot reconciliation cron - syncs currentinventorysnapshot with scan truth for active tag_listing items - emails drift report via googleMailSender to sdtech - runs daily at 5:30 AM, CLI trigger: --reconcileInventorySnapshot |
|
| 36361 |
136 d 7 h |
amit |
/trunk/profitmandi-cron/src/main/java/com/smartdukaan/cron/scheduled/ |
Switch @Scheduled updatePartnerLimit to batch-tracked version. Was calling legacy scheduledTasks.updatePartnerLimit (one synchronized tx, writes to all ~1500 partners each run, no audit). Now calls batchScheduledTasks.updatePartnerLimitWithBatch (writes only changed partners in per-partner REQUIRES_NEW, records each run in cron_batch / cron_batch_item, sends failure email on partial failures). Same cadence (every 20 min), same business logic. Runs are now visible in /admin/cron-batches UI. |
|