Jev, Decision Models: Replacing a Chat Model with a Multiple-Choice Scorer - the OMR-Sheet Analogy, State / Question / Choice / Noul / Score, Cricket-DRS Human-in-the-Loop Tiers, a Live 500-Email Race, and Where It Earns Its Keep

Sugam opens with a blender ticket ('third time the blender arrived cracked - send my money back and close my account') and has the cohort vote in chat: refund, cancel or complaint. Then two friends: ask one 'is it raining?' and get three paragraphs ending in 'yes'; ask the other and get 'yes, 92 percent sure'. The second friend is Jev, a decision model: you give it a case and a fixed list of answers, and it picks one and reports how sure it is; it cannot write a sentence. The analogy he uses for the whole session is the OMR (optical mark recognition) answer sheet used in competitive exams: a chatbot is the long-answer paper, Jev is the sheet where the options are printed beforehand and you fill one bubble with a certainty next to it. A short game builds the habit: farewell message for a colleague (chatbot), is this product review angry yes/no (Jev), summarise a 40-page policy PDF (chatbot), route a ticket to billing, tech or sales (Jev). Rule of thumb from the PDF: if the answer fits in a bubble use a decision model, if it needs a sentence use a chatbot. He places Jev in a small family he calls system-one models: Laya (an open-source Indian-origin project; he found it less accurate on his data but had not finished comparing), OpenAI's own decision API (not yet in India per him) and TypeSafe's Jev.

Sugam's mechanism is the difference between typing and ticking. A chat model generates token after token and charges for each; Jev reads the state and scores all options in one pass, writes no words, so output tokens are free and the cost is input only. The takeaways PDF puts vendor numbers on it: roughly a quarter of a second per call (70-500 ms), 0.042 USD per million input tokens, launched 15 September 2026 and still in early access. How it scores internally is a black box, he says: TypeSafe has not published a paper, so the cohort should not claim to know. His own ledger: the old chat-model triage cost about a thousand dollars a month; with Jev it is cents ('maximum a dollar or so a month' for the routing decision; the reply drafting still uses a small chat model). In his live race the cost comparison is shown as multiple orders of magnitude (the chat run printed four zeros of cost against two for Jev) and in the PDF's illustration, at a million emails with an assumed 500 tokens in and 100 out, about 10,000 USD for a chatbot against about 21 USD for Jev, and about 417 hours against 69 hours one after another. He repeats the honest caveat: those are illustrative; measure on your own data.

State is the case file the model reads - the email, the order, the chat; imagine the document you would hand a new joiner. Question is the exam paper: each question has a name you pick and gets one answer back. The types: Choice picks one option from a list (billing, tech or sales?) and returns the pick, odds for every option and a confidence; Noul (the PDF's spelling; heard 'now' or 'null') is a yes/no that returns one number, 1 for yes, 0 for no, 0.5 for unsure ('does the customer want a refund?'); Score places the case on a scale you describe (calm, annoyed or furious?) and returns a position, odds and confidence; Box, a newer sixth primitive he mentions, takes an image and a question and answers where or whether something appears (his example asks whether a cup is in a photo). The TypeSafe playground shows the same fields: the state, the question, the instruction and the criteria. In his repo the call lives in lib/models/jev.ts, which builds the payload from the task definition and reads back the choice, probabilities and cost. He notes a non-coder can simply tell their coding agent to implement Jev for a named situation and it will pull the SDK and wire the call.

Cricket's Decision Review System is Sugam's design pattern. The on-field umpire (Jev) decides almost every ball in seconds with a soft signal; a close call gets a review where the AI suggests and a person confirms with one click; the very close ones go to the third umpire, a person deciding without the AI guess shown. In config/tiers.yaml the thresholds are confidence 0.90 and up (software routes automatically), 0.60 to 0.90 (review) and below 0.60 (a person decides); the PDF later recommends starting stricter, 0.95 and 0.75, and relaxing only when calibration earns it. Stakes change the thresholds: would you let a 95-percent-sure AI auto-approve a 200-rupee refund? A 20,000-rupee one? His production rule is that refunds never auto-reply: the system fetches order history, drafts a private note for the agent, and a person approves before anything is sent. The approval hierarchy can have more tiers (a manager for a 20,000 refund). He ties the review step to evals: user queries go into Langfuse so he can build golden examples and label them, and every person decision becomes a new labelled example, closing the loop. A gotcha from the PDF: confidence is not the top probability; with three options it is (top - 1/3) divided by (2/3), so a 71 percent favourite is only 0.57 confidence. He adds a rule of thumb: the more it hurts when wrong, the higher the bar.

The demo repo (github.com/Sugamm/jevai-outskill-session-demo) needs two keys, a TypeSafe key from console.typesafe.ai and an OpenRouter key, and is a Next.js app run with npm run dev; npm run smoke tests one email, npm run dataset and npm run review generate and check your own labelled set. Four rules shape the brief: answer only from the list, know when unsure, cost in cents, prove it on your own data. The exam paper is config/task.yaml: the question ('what is the customer asking for in this support email? if more than one request, choose the most consequential: cancel account over refund request over complaint'), then a one-line definition per option - complaint (reports a problem, no refund or closing asked for), refund request (asks for money back) and cancel account (asks to close the account or membership). That tie-break line is the lesson: the blender ticket that split the room split it because nobody had ordered the options. Data lives in data/: about 500 emails sourced from Kaggle (an electronics store) with a hand-made golden label, a difficulty tag and a reviewed flag, ids like e500; the PDF tells builders to bring 200-500 real labelled examples and start strict. He keeps the answer key because he never trusts a vendor benchmark he has not reproduced ('every mango seller says his mangoes are the sweetest - taste them on your data'). The reply itself, for his production flow, is written by a separate small chat model (GPT-4o mini) after Jev decides.

In the /race page both models get the same 500 emails, the same three options and a concurrency of ten requests; Jev finishes far faster and cheaper (he cites 13 seconds against a minute for the chat run and a cost gap he describes as the 'mic-drop moment'; the on-screen cost figures are partly unreadable in the audio). He explains p95 latency (95 of 100 users see at most that latency) and then moves to /calibration, the honesty test. Three tickets: a toaster stopped working with an order id (both models say refund; the chat model is 55 percent sure, Jev 100 percent); 'the app keeps crashing, honestly thinking of leaving' (both say complaint, borderline, so the tier asks for a review and the tech team is the real destination); and the blender ticket 'send my money back and close my account' (both say cancel; Jev 95 percent, the chat model 25 percent - and he calls it wrong because the official label was refund request, a reminder that Jev can be confidently wrong and that his own cancel tickets were landing in the tech bucket). Break-it probes follow: sarcasm ('oh great, another cracked blender, love that for me') - both complaint; 'refund the blender and also close my account' - both cancel, 99 versus 60 percent; a Hinglish refund demand - both refund, 100 versus 50 percent; an arithmetic ticket (charged 1,200 then 2,400, refund the difference) - refund. The PDF's honest reading: accuracy can trail a big chat model, which is why uncertain cases go to people, and results shift with how options are named. He also says he intended to compare against GPT-6 but his team restricted large models on the shared key, so the demo used a cheaper chat model.

Sugam shows Outskill's Freshdesk with real tickets (he apologises for the risk) and explains the production split. Jev decides the intent and routing (about 80 percent of tickets); a separate small chat model (GPT-4o mini) drafts the reply from a knowledge base; the draft is posted as a private note, not sent, so a support agent approves it. The example is a student whose family member died asking to reschedule the remaining days of a 14-day programme; the system returned 'insufficient KB' because the sender's email address did not match the registered one, then drafted a sympathetic reply asking for the registered address. Refunds and commercial questions go to humans; schedule and session-detail questions are fully automated. A QA developer reviews tech-bucket tickets and improves test cases. He keeps the draft-only mode until he is certain the knowledge base is complete, using the progression wheel, skateboard, bicycle, car. Pending note for the cohort: he also wired DevOps alerts to Slack and a phone-call agent, and mentions a hiring CV screen, covered in the use-case concept.

Sugam asks the cohort for their industries and walks examples, all with the same shape of cheap decision plus a person on the borderline. Fintech: is this UPI payment suspicious - allow, OTP check, block; a person steps in when blocking would hit a real customer. E-commerce returns: instant refund, pickup, inspect; a person on costly items or repeat returners. Insurance: fast-track, review, investigate; any rejection goes to a person. Hospital front desk: routine, this week, today. DevOps: page now, ticket or ignore; the third umpire is whether it wakes someone at 3 a.m., and he has a Slack bot and a VAPI phone call for payment and lead-generation outages. Hiring: yes, maybe, no on a CV, a person on any no. Social apps: allow, review, remove. EdTech: FAQ, teaching assistant or teacher; a person when the student is stuck twice. Longer stories: satellite and drone imagery for construction progress and crop classification (his SatSure experience, selling yield data to lenders), and a marketing-attribution product (omnichannel ad data) where Jev can judge whether a design or copy needs changing; he also notes AEO/GEO as a market. For engineers, the bouncer: put Jev at the door of a coding agent so each action gets two yes/no questions - does it destroy or overwrite anything, and did the user ask for it - and also use it to turn many filters into a structured database query. Three tests (screenshot): can you print the bubbles in advance, does it happen hundreds of times a day, and can a wrong bubble be caught? Three yeses, build this week; two, build it with a person on the uncertain cases; one, use a chatbot or person. Fine print from the PDF: independent papers found chat models more accurate on complex reasoning, option names change results, the biggest speed and cost multiples come from TypeSafe's own tests, early access pricing may change and TypeSafe cannot yet prove it is not subsidised. Homework: pick one decision from your own project, run the three tests, then edit task.yaml, data/emails.jsonl and tiers.yaml. 25 community Jev projects are listed in the PDF (links to be added).

Shivani tells the cohort that this is the second-last weekend: next week's session is the town hall (fill the form in the WhatsApp group to present; people who presented before and have made no major change, or have not monetised, can skip), with Harshit Yagi expected to guide the product walk-throughs. The Sunday session is the final one; the shared Sugam document will be posted on the LMS and in WhatsApp.