Subversion Repositories SmartDukaan

Last modification

• Rev 37472 – 2 d 7 h
• Author: amit

• Log message:

  IMEI activation: one snapshotted daily pass per brand, on one thread, with per-brand metrics
  
  Four @Scheduled jobs every 5 minutes become two daily passes. Oppo and realme
  share one thread and alternate in 25-imei chunks, so exactly one ChromeDriver is
  alive at a time instead of four; vivo keeps its own thread since it is direct
  HTTP and does not contend for a browser.
  
  The pass snapshots its pool before any browser starts and walks that list to the
  end. It never re-queries, and that is the actual fix. A failed lookup never
  reaches dateMap.put, so no row is written, so createTimestamp is not bumped, so
  the imei was eligible again on the next tick five minutes later. Measured 29-Aug:
  realme issued 4,524 requests against 1,004 distinct imeis -- 4.5 asks each, 78%
  of the day's budget spent re-asking -- while oppo, which rarely fails, sat at
  1.03. More requests hardened the block, which caused more failures. A pass bounds
  that: a failure costs one retry tomorrow, never one in five minutes.
  
  This supersedes the r37447/r37448/r37449 argument about driver count, which was
  about the wrong variable. That argument blamed realme's collapse on CPU
  contention pushing the captcha render past the element waits. The logs do not
  support it: on 29-Aug oppo took ZERO canvas timeouts across all 24 hours on the
  same box, same six cores, same driver count, same captcha vendor, load average
  0.9 -- including the 15:00-23:00 window in which realme solved nothing at all.
  Realme's own canvas wait is 15s against oppo's 8s, so the longer wait is the one
  expiring. What realme's timeout rate tracks is its own daily request volume, and
  it resets at midnight: 920/day -> 0.3%, 3,467/day -> 28%, 4,524/day -> 75%. That
  is realme.com declining to serve the widget.
  
  DAYS=0 is deliberate and is not an off-by-one: the pool filter is
  createTimestamp < now().atStartOfDay().minusDays(DAYS), so DAYS=1 measures
  against yesterday midnight and silently yields a two-day cadence, which is what
  oppo and realme were running.
  
  Sizing measured on prod for a midnight start: oppo 4,133 and realme 2,118 imeis,
  11.7h + 8.4h = 20.1 hours of a single thread. It fits with no slack; if the
  'pass finished' counts come in short of 'pass starting', the lever is DAYS=1
  rather than a second thread.
  
  Observability: ImeiActivationGauges publishes the funnel per brand on
  /actuator/prometheus, which alloy already scrapes on this host -- due, churned,
  captcha_shown, captcha_solved, answered, dates_found, errors, run_seconds and
  last_finish_epoch. Each stage fails differently and says what broke. Rates are
  left to PromQL. The stage that matters for health is answered: churned>0 with
  answered==0 is precisely the shape of both silent outages this year (oppo wrote
  nothing for a week; the vivo captcha solver was dead for 46 days). dates_found is
  deliberately NOT a health signal -- when the multi-year backlog drained at the
  end of August, yield fell from ~100% to 2-3% on the same day across all three
  brands with nothing broken.
  
  Nagios cleanup: the Nagios server and every NRPE daemon are gone, so
  WriteToPropertiesFile and the commented-out blocks that fed
  nagios-cron.properties are deleted, and NagiosMonitorTasks is renamed
  BalanceMonitorTasks for the transport it actually uses. Noted there that nothing
  calls it -- there is no @Scheduled entry and no other caller -- which is why both
  balance gauges have always read -1.