It's been a very long time since I wrote one of these. I was finding some flow and focus. But I want to pause and take stock—because what I'm feeling right now deserves to be expressed. Think of this as one specific denoising step in the diffusion process of figuring out who I'm becoming. A step that brings some clarity.

The Landscape So Far

So far, I've explored various planescapes—or maybe the surfaces—of machine learning paradigms. Right from statistical learning to probabilistic machine learning, causal machine learning, interdisciplinary areas like machine learning for dynamical systems, LLMs, diffusions, generative models, reinforcement learning, and more. And when I connect what we really learn versus what exists in the industry, I can see a lot of differences.

Especially when I was in India working full-time, I never worried about or optimized myself against a specific role, because it was always changing. In one company, the data scientist would be doing anything possible or anything required with data—right from very simple dashboarding to working on reinforcement learning problems, causal inference problems, and computer vision problems. That's how my career was.

At Mu Sigma, I got to work on regression and mixed-effects models with one particular client. For, say, six months, it was completely deep-diving into that. On an everyday basis, we'd be working on it. The problems were not well-defined, and the data was very messy. And I don't agree with the convention that data science is always 80% data preparation. That's a leftover from Kaggle competitions and boot camps. Even in my initial projects, we spent quite a lot of time on hypothesis testing, running different types of statistical models, benchmarking them, and presenting to stakeholders. I would say it involved frequent experimentation—exploring multiple possible solution spaces, validating them, presenting to stakeholders, and then holding discussions at multiple levels, with seniors and with the business, about what would be the optimal approach.

Agent-Based Modeling and the Gift of Complexity Science

For one specific project on agent-based modeling, the concept was not taught anywhere. You couldn't really find an online YouTube tutorial to learn it. There might be some, but they're very research-experimental—simulating artificial life, artificial societies, or maybe simple crowd simulations. You can find tutorials for Mesa (M-E-S-A) and similar software. But in an industrial setup, in real-time problem-solving, agent-based modeling was a new topic that gave me a very good introduction to complexity science within just three or four months of my full-time role.

And it still sticks with me. Complexity science gives you a very good understanding of how things from subatomic particles to bigger communities, countries, and societies share common natures—interactions, emergent behavior, nonlinearity, and such things. It gives a different way of seeing things.

Bayesian Networks and Simulation at Scale

Then I got to explore Bayesian networks—how you come up with the posterior, how you derive the conditional probabilities. To construct a Bayesian network, one needs both business understanding and statistical validation. For example: can we say demography influences brand preference? Or does age have an impact on incidence—where incidence means the likelihood of a customer wanting to drink or consume something at a given time?

Such a network could represent the consumption of a specific product. Take an hour of the day: it's currently 9:00 p.m. in New York, I'm a student, I'm from South Asia, and so on. What is the likelihood that I'm currently drinking, say, milk, or one of a number of options? Let's say the company wants to understand the likelihood across some 200 SKUs. What's the likelihood of a particular brand? What's the likelihood of me drinking Dr. Pepper at this particular moment?

For a specific person with a set of demographic, psychographic, and socio-economic attributes, given a particular day and season—imagine you want to simulate this for a sample of 10,000 or 100,000 people, and you want to understand the patterns that emerge through the simulation.

This involves a lot. From the research side: how do you understand Bayesian networks, design them, validate them, experiment with different hypotheses, and construct a better network? And how do you validate it with the macroscopic data you have? On the other side, we have to write the code, build the simulations, and run them on, say, a Scala or R distributed engine. These are the kinds of problems I've worked on.

A Career Without Role Boundaries

Once a project reached a certain point, I'd move on—to a computer vision problem, or optimization, or discrete event simulation, or multitask learning, or coming up with a custom deep learning architecture for physics-informed neural networks.

My career—and my perception of whether a role is data science, machine learning, AI engineer, or machine learning engineer—has been shaped differently from what you see on the internet, YouTube, or communities. People always say things like "data scientists don't deploy models" or "machine learning engineers only deploy models and scale them." Those were very limiting conventions. That's why I never tied myself to any specific role. My first role was decision scientist, the second was data scientist, and the third was machine learning engineer. I got to work on a range of problems. To be honest, I've explored a very good set of problem statements—worked on them, built things, and deployed wherever possible.

The American Job Market: A Different Beast

When I came here and looked at the descriptions across different companies—and I've written about this on LinkedIn—the data science roles are not that technically heavy or quantitatively demanding. They're mostly just number crunching. Very specific companies have specialized roles like data scientist for marketing science, data scientist for supply chain, or data scientist for causal inference and experimentation. Such roles demand niche skill sets. But a generalized data science role is just working on some product, getting the numbers, building dashboards, and answering questions. It's in no way leading toward working on machine learning solutions, statistical solutions, or prescriptive analytics—building models from an optimization point of view. I couldn't see that across the jobs I applied to.

As for machine learning engineers, the descriptions vary. Then there are new roles like AI engineer, agent developer, or technology advisor, where you're developing AI prototypes. But I'm not really excited by such roles.

Two Roles, One Common Foundation

What I'm feeling—what I'm becoming—is what I'm trying to express here. I see myself aligning with two specific roles. One is applied scientist (or applied researcher). The other is research engineer.

I've always been interested in research scientist roles, but considering the PhD requirement and the deeply focused research experience needed, I'm kind of far from that right now. But in the near future, what I can optimize myself toward are these two roles: applied scientist and research engineer.

One might argue that both roles require different skill sets, but when I read through the descriptions and try to understand the crux, I can see that both are built on top of very common skill sets and common expectations: bringing research to scalable experimentation or scalable solutions.

In many places, research is conducted on very specific small-scale datasets or under confined assumptions. They might be trained on very large datasets, but they might want to scale further, try multiple options, and so on. The speed at which frontier labs are producing research and delivering consumable products—with strong research focus—is because they're able to speed things up at scale. They can consume a lot of compute, scale things, and run them in parallel.

The Interdisciplinary Edge

Where I find myself is that I'm able to understand the fundamentals of a specific concept and see where we can bring it into real-world use cases, as well as think from an interdisciplinary point of view. It's not just, "Oh, I learned LLMs, I learned KV cache, I'll just implement it," or "I learned speculative decoding, I'll just work it out on a small scale." That's still one style of deliberate practice. But where I see myself is more in terms of how you can extend research to other aspects, or how you can try to use it for other use cases.

For example, someone learns about normalizing flows. How can we use it for representation learning—specifically in single-cell data, genomic data, or recommendation systems?

How do you think in an interactive way—interaction between two disciplines? I learn something, I know how certain things work, and I also know some problem exists in a specific area that needs a specific focus.

Take recommendation systems. Even if you look at Twitter's recommendation algorithm, which was open-sourced months ago, it's essentially matrix decomposition or learning the likelihood of liking, sharing, reposting, and such things. But it doesn't learn the latent expressions of a specific user—what they're really trying to do with those interactions, what they really think about a given feed. This was discussed in a paper on the inversion problem, which connects the problems in machine learning and social media algorithms with the behavioral state of mind or the objectives of consumers—what they really want or think. It's about what these algorithms need to satisfy but fail to do.

Can we think in terms of causal representation learning? Can we think about how to bring System 1 and System 2 thinking into social media algorithms themselves? Coming up with such hypotheses—and if they offer good exploration ground, going in depth—is what excites me.

I've also started developing a niche interest in causal inference, probabilistic models, and learning theory—from Vladimir Vapnik and VC dimension onward.

The Knowledge Graph in My Head

Where I see myself is that I can quickly grasp things because I don't need everything laid out A to Z. It's more like link prediction—interconnecting things. For example, I literally learned how cognitive systems work fundamentally—how System 1 and System 2 help us, how the hypothalamus, hippocampus, and neocortex complement each other in terms of learning. When I encounter a new concept in continual learning, I can quickly connect things: "Oh, this is how they're also connecting." And when I connect back with some residual knowledge on machine learning, I can see how things are working.

World models are a great example. Grasping and learning about world models was very comfortable for me because in the past I'd learned and implemented system identification and model predictive control. Over the years, I've been building models on enterprise and business operations—which is essentially identifying a specific system. In many projects and presentations, I've approached things from a system dynamics or systems thinking point of view. I come from a control systems background, and I've adopted systems thinking at various levels. I also read Thinking in Systems by Donella Meadows.

It's given me a kind of mechanistic view from the beginning—seeing how data problems arise from some underlying mechanism or system. That's what causal inference and structural causal models are about: there is some mechanism underlying. The representations and their behavior might differ, but they all come from the notion of underlying systems or models.

When I connect world models with system dynamics, and system dynamics with cognitive science, it's all talking about the same thing: how a machine, system, or engineer tries to understand a system in a very mathematical or representational way so they can interact with it.

That's what The Nature of Explanation discusses—the brain has a model and is able to run predictions or form hypotheses. The same thing applies to structural causal models: when you have an SCM, you can come up with different potential outcomes. Similarly, with a world model, you can use it to plan—because you can simulate different outcomes for different actions. For an action and an outcome, you can understand the cost of doing that. Based on that, you can plan or optimize.

Over time, things become a kind of ontology or knowledge graph where you can quickly navigate the interconnectedness. It empowers the underlying mental model and first-principles thinking. It keeps asking: why is this working, or why is it not? What could be the assumptions behind it? What are the mathematical foundations supporting it? You can quickly validate, navigate to the resource, spend time reading and absorbing it, and then develop from there.

The Right Direction

If I really visualize all of this, I can see that moving toward applied researcher, applied scientist, or research engineering roles is the right direction for me—better than ML consultant or data science roles focused on number crunching, or machine learning engineer roles focused purely on software and coding.

Those roles don't give space for thinking at different layers—thinking abstractly, thinking in detail with mathematical or algorithmic rigor, or thinking in systems. I don't think that's available in many machine learning engineering roles, where they typically ask you to work on refreshing data or iterating on a specific thing.

My understanding of these roles—applied scientist and research engineer—is clear: take from an idea to execution. From theoretical validation, gathering references, putting together a blueprint, experimenting, executing, and seeing where you've landed. It's more or less like venture-building around a specific problem statement.

That's what I've been doing in the past. It wasn't always the case—sometimes the problem statements were underwhelming. But most of the time, I was touching a fresh problem from a very chaotic definition and then keeping on defining and understanding it until arriving at a specific point to go from there.

It's not like someone comes from a Jira board and says, "Hey, this month our target is to optimize this particular model to reach this reward." And you just play around with different things. I don't really like that—though in the future it might be necessary, and I should be open to it. But I'm expressing what the ideal or exciting role looks like to me.

Where I'm Building From

I think this should be a good justification for doing a master's and putting in all this effort. In the American context, or in the contemporary Gen AI and frontier landscape, my past work would be considered applied data scientist or applied machine learning engineer experience. But now, the question is: in this contemporary frontier machine learning landscape, where do we want to be?

Should we take on support or maintenance of machine learning models with minimal adjustments and minimal contribution? Or should we be at the point of execution and scaling of interesting problem statements? The interesting problems could be anywhere—building causal models, building probabilistic models for marketing teams, or in generative AI, building world models or models for prediction markets.

I see it as: how can we use machine learning to solve a problem? In that case, it's not just taking a framework or library and applying it. It's about how you take a business problem into a machine learning problem, and how that machine learning problem can be approached with multiple layers of thought—abstraction, theoretical or mathematical thinking, systems thinking, and algorithmic thinking.

Honest Gaps

I've also understood where I lack. I lack in terms of coding—specifically GPU programming and distributed computing. I believe I can translate mathematical concepts into code. For example, if I have an algorithm for some optimization problem, how can we implement it? Or say we have a probabilistic graphical model—how can we convert it into a factorized posterior and then get to the inference side of things? I've been exploring all of this and have used it in the past, but it still needs more practice so that I can develop strong muscle memory. It's not that I can't solve it, but to an extent, I need that muscle memory to build—not just mental models.

The other area is having more conviction and understanding of quick mathematical validations on the probabilistic, algorithmic, and statistical side.

Compounding, Not Scattering

I believe I've laid out a platform through this master's, through the last two semesters—done enough exploration, worked on problem statements for my projects and personal interests. Giving it shape, making it focused, and compounding is now very important.

One thing I lack is compounding on a very specific area. Running behind multiple things or working in industry might not give that leverage. The knowledge has compounded, but industry might value compounded work—proof of work, of sorts.

Becoming a research engineer, or grooming myself in research engineering, requires a lot of deliberate practice, iteration, learning, focus, and compounding. That's what I'm trying to build and train myself on.

I don't think I should be preparing for interviews. It's more than that—it's about exploring as much as possible, executing on a different level to prove that we are really purposeful toward research engineering, rather than just maintaining surface-level relevance like, "Oh, I know AI, agents," and such things. I don't find that useful or productive.

The industry or the internet always comes up with some random or very confined view on career building, because nobody has explored all the possible things—they just share what worked for them. We can't stick to somebody else's roadmap. It needs to come from our side—this is what our mental model and long-running learning has helped us explore.

That's why I'm trying to put everything together in a focused way—all the curricular, co-curricular, and extracurricular learning—to help me do something useful on the frontier of machine learning and AI, rather than running behind some unknown or illusory, undefined target.

The Value of Real Problem-Solving

At the same time, internship experiences would be really helpful because they provide mentorship, a platform to get validated, constructive feedback, and exposure to new problem statements.

It shouldn't all be done in imagination, and it can't always be just, "I did some projects." It has to happen through real problem-solving. Internships provide good motivation and everyday wins. They give feedback, mentorship, and motivation through everyday wins.

Sometimes it feels weird and complicated being a student after six and a half years. But I feel it's very good, because since the degree started on August 21st, and now it's March 25th—in the last seven months, there's been a lot of learning in terms of how to think through machine learning and how to pick up a research problem.

I have a lot of projects piled up—code and reports—but publishing is yet another task that needs additional effort. I'll be doing all that: working on papers, working on frameworks, building small research prints, working on exercises. That's what I'm going to focus on. I'm going to use social media just to share progress and nothing else.

The Four Problem Statements

Currently, for this semester, I'm working on four problem statements:

  1. Causal representation learning
  2. Continual learning and self-supervised learning
  3. World models and model predictive control
  4. Energy-based modeling for dynamical systems

These four are interdisciplinary, with space for complexity science, cognitive science, and causality. I'll also be revisiting fundamentals. It's going to be on the trifecta of fundamentals, frontiers, and frameworks.

This is how things happen. Through course lectures, homework, and self-study, I've explored and confined the research surface or experimentation surface. Now I feel quite clear about what I need to achieve, what I need to do, get results, and what I need to prove through each problem space. I'll be focusing on that and also want to explore and write more in the upcoming notes.

Public Display of Accountability

I believe I'll be able to get to work on frontier machine learning and AI problems. Let's see how far I go. I'm treating this as PDA—public display of accountability. I can look back and see where I've been, whether I've been distracted or focused, and where I've reached. I have a good two semesters remaining to focus on this.

I'll utilize the learning from the past—from different coursework, interactions, and learning from different platforms, researchers, and engineers on the internet. It's going to be a combination of craft, experimentation, and trust.

The combo and effect will work, because sometimes I've had this issue with instant gratification—doing experiments and expecting something to happen, but it doesn't.

The Hard-Earned Lesson

I've understood that nobody would provide instant gratification. When I was working, if I did something, I'd get appreciations or a feeling of a win. But one thing I've understood over time is that appreciation or wins don't need to come from the external world. They can come from our internal world. "Okay, we've progressed from here to here." As simple as that.

That's a hard-earned lesson. You might think, what's the reason behind someone writing on the internet? Is it just a reference? No. There might always be an expectation: "I'm being resourceful, I'm showcasing what I know, what I've solved." And there'll be an expectation that people might read it and share feedback.

I've realized that's not happening. Sometimes you can't even expect your own peers to appreciate it. That's fine—there are alignment issues and various factors. But the hard-learned realization is this: appreciation, feedback, critique, or confrontation—everything is only true if it comes from our inner self.

So I'm going to trust my inner self to be my greatest mentor.

A Note on Confidence and the Dunning-Kruger Curve

One thing that's always in my mind is the Dunning-Kruger effect. Most of the time, people use it to comment on others—"Oh, they're in their Dunning-Kruger phase," or "They're feeling overconfident beyond what they're really capable of." That's all fine. But here's the thing: if the Dunning-Kruger curve applies to everyone, then without passing through the initial phases, one cannot reach the valley of despair, then the slope of enlightenment, and finally the plateau. The curve is going to be there for all of us—but it plays out in a very distributed way.

For specific concepts, I might be in a state of too much confidence. In some places, I might be at the stage where I'm just starting to make sense. But right now, I feel like in most areas I can make sense of things. That's what I described earlier—the knowledge graph, the interconnected way of learning. It helps me understand.

And obviously, I'm not thinking this is going to be easy. That's exactly why I'm saying it needs continuous exploration. It doesn't just end with one or two implementations or a project. It needs to be continuous. That's why even while doing the master's, I'm taking courses, setting targets, and reflecting on them—because I've never felt that this is easy or enough.

Sometimes we talk confidently. When somebody is talking confidently, that doesn't mean they feel like they know everything. It means they have the confidence to say, "I can learn this, I can figure this out." And also, it's confidence that comes from knowing where we lack and where we need more effort. It's not blindly climbing things.

The Roller Coaster of Learning

When we get into something—a process of achieving a goal—there will be a lot of ups and downs. It's a roller coaster. Some moments add momentum: "Oh, I've implemented this and it was quite manageable." Or when you learn something new and it makes other things click.

For example, when I was learning about diffusions and flow-based models separately, it was difficult to understand. But when I stepped back and deep-dived into energy-based models, things started connecting. Where energy functions, scores, diffusion processes, and flow-based density estimation all sit in relation to each other—the interconnections between learning the density versus learning the scores—suddenly made much more sense.

And when you have this layer of denoising in your own learning process, you're able to proceed further. Each concept you truly internalize doesn't just help with that concept—it retroactively clarifies everything adjacent to it.

So yeah. It's a continuous process. Let's do it. Let's focus. Let's continue—and we'll see how it goes.

This note is part of my ongoing public journal—a way to track my own diffusion process from noise to clarity. If you're reading this and walking a similar path, I hope it helps you feel less alone in it.