another post about "chatgpt changed my analysis workflow."
another screenshot of claude writing complicated sql. and you're sitting there with your 6 years of experience wondering if your tableau dashboards just became worthless.
let me tell you something. using chatgpt to clean data doesn't make you ai ready. asking claude to write your sql queries doesn't either.
everyone thinks that being an analyst in the ai age means learning more tools. cursor for sql. julius ai for visualization. claude for analysis.
wrong. it means understanding 3 things.
1/ which problems actually matter in an ai world.
2/ which metrics make sense for probabilistic systems.
3/ which insights drive decisions when the ground truth keeps shifting.
tools are commodity. your judgment about what to measure isn't.
you're a bridge builder. business has questions about ai systems. data has answers hidden in token logs and confidence scores.
you build the bridge between "why is our chatbot weird today?" and "the model's perplexity increased 23% after yesterday's update."
catching distribution drift before customers notice
quantifying the cost of hallucinations in rupees, not percentages
trust built = explainability × consistency × communication clarity
making black boxes slightly less black translating ml engineer speak to ceo speak
see the shift folks? most analysts optimize for the old world. historical accuracy. pretty visualizations. statistical significance. but the highest leverage is in the new spaces. uncertainty management. realtime evaluation. trust quantification.
two games, different proof
depending on whether you want to crack a role in internet first vs ai first companies, your problem statements will change. a lot.
1/ internet first companies? swiggy, paytm, phonepe, dunzo. these companies have data problems you know. conversion funnels. user retention. revenue optimization.
their analysts need to show how ai amplifies existing metrics. reduce cart abandonment using predictive models. increase ltv with personalization. optimize delivery routes with reinforcement learning.
the math is familiar. the tools are just more powerful.
2/ ai first companies cursor, openai, bolt, replit, etc. different game entirely. the product IS the model. no model, no company.
their analysts need to measure things that don't have precedent. how do you measure conversation quality at scale? what's the right metric for multilingual performance? how do you catch model degradation before it ships?
the math is unfamiliar. the tools don't exist yet. you build them.
building proof that matters
6 steps to build proof of work.
i’ve covered both, ai first companies and internet first.
what doesn't work
tool obsession "i know langchain, llamaindex, weights & biases, mlflow..." great. what insights have you delivered? what decisions have you influenced? what money have you saved?
tools and all is okay if you have less than 4 years of experience. not beyond that.
analyst role is being split into three
type 1: ai system evaluators they sit with ml engineers. design evaluation frameworks. measure model behavior. quantify uncertainty. create trust metrics.
these analysts will thrive. ai needs evaluation more than ever.
type 2: decision scientists++ they partner with product. connect model metrics to business outcomes. design experiments for probabilistic systems. basically analysts who understand ai deeply.
these analysts will evolve. traditional + ai skills = lethal combination.
type 3: report generators they pull numbers. update dashboards. create weekly reports. answer ad hoc requests. basically human sql interfaces.
these analysts will be automated. ai will get better at writing sql than most analysts.
harsh? look at the job postings. "must understand transformer architectures" for analyst roles. "experience with llm evaluation" required. "statistical methods for probabilistic systems" mandatory.
the writing is on the wall. or should i say, the tokens are in the context window. (sorry for the dad joke)
your move
in an internet first company? stop using ai just to work faster. find one growth equation lever. show how ai can improve it 30%+. connect to revenue.
want a role in ai first org? pick a measurement problem that doesn't have a solution yet. build one. even if crude. show you can think in probabilities.
if you want to build bridges find a company using ai badly. show them what they're measuring wrong. build the right metrics. become indispensable.
pick one follow my steps and build something that solves it. ship it this weekend.
it does not need to be perfect. trust me. doing a failed attempt at this will be better than 99% of others who have never even tried this.
because in 12 months, "proficient in sql" will be table stakes. everyone will have ai assistants for that.
but "built evaluation framework for code mixed language models"? "reduced inference costs by 40% through smart sampling"? "created early warning system for model drift"?
that changes orbits.
when ai can analyze data in seconds, clean datasets in minutes, and generate reports instantly, what's left for analysts?
the same thing that was always most valuable
knowing what questions to ask about systems no one fully understands yet.
