Subversion Repositories SmartDukaan

Last modification

• Rev 37788 – 12 d 9 h
• Author: amit

• Log message:

  activation: stop a broken JVM from spending the day's imei pool, and take the
  selenium atom read out of the nested jar
  
  Oppo/Realme activation collapsed on 21-22 Sep. Every WebDriver command failed with
  java.util.zip.ZipException reading a Selenium JS atom:
  
  W3CHttpCommandCodec.amendParameters:227 -> executeAtom:397
  -> com.google.common.io.Resources.toString
  -> org.springframework.boot.loader.jar.ZipInflaterInputStream.read -> ZipException
  
  Scale, from fofo.activated_imei: a ~30-50 errors/day baseline became 17,508 on 21-Sep
  and 27,768 on 22-Sep. On 22-Sep it touched 3,920 realme imeis for 0 dates and 4,824
  oppo for 96, against a normal 76-100% hit rate. Realme burned its entire day pool by
  11:07 and then correctly went quiet, having answered nothing.
  
  TWO INDEPENDENT FAULTS, one fixed each way.
  
  1. The pool was spent on an outage. restUnanswered stamps every unanswered imei so the
  20-second tick advances instead of re-handing the same rows -- correct for a per-imei
  failure, catastrophic for a systemic one, because a stamped row does not come back
  until its rest expires. So a JVM that cannot read a jar quietly consumed a day of
  payout data. Now: if NOTHING in the chunk was answered and the chunk had more than one
  imei, that is infrastructure rather than a verdict, and nothing is stamped. Cost is a
  re-ask of the same chunk next tick -- loud and self-limiting -- instead of the day.
  Same class of bug as the carlcare transient refusal (r37537): far-end/our-end noise
  must never be recorded as an answer.
  
  2. The read itself. Selenium loads its atoms as classpath RESOURCES on essentially
  every command, which inside a fat jar is a nested-jar read on the Spring Boot 2.0.2
  (2018) loader. Three different inflater errors appeared on prod -- "invalid stored
  block lengths", "invalid distance too far back", "invalid code lengths set" -- on a jar
  whose outer AND extracted nested archives both pass `unzip -t`. Intact bytes with three
  distinct inflater failures is a reader fault, not a file fault. bootJar now sets
  requiresUnpack for selenium-remote-driver, so it is extracted to a real file at launch
  and the atom read never touches the nested reader. Verified in the built jar: the entry
  carries UNPACK:<sha1> and is STORED rather than DEFLATED.
  
  TRAPS WORTH RECORDING.
  
  - A restart is NOT a diagnosis here. It was restarted 13:16 on 22-Sep and still failed
  for three more hours at ~1,260/hour, then a 16:36 restart came up clean -- same jar,
  mtime unchanged. Anyone reading "restart fixed it" should distrust it.
  - Concurrency alone does not explain it: up to 2 scheduler pools ran these tasks per
  minute in the broken window AND in the healthy one.
  - The GlitchTip board under-reported this badly (#1495 lastSeen 17-Sep while the log
  held thousands on 21-22 Sep), so the board is not a reliable outage signal for cron.
  Judge this lane on dates written in fofo.activated_imei, not on issue counts.
  - Deploy the cron jar stop -> replace -> start. The jar on disk was overwritten in place
  at 12:31 on 21-Sep while a JVM held it open, which is how this started.