In this blog, we will see what a complex system is, why we need systems thinking in the age of superintelligence for solving high-stakes and consequential problems, and why forward-deployed engineers need it beyond just doing integrations and tuning.

As I write this blog, on one side, 1. Dario Amodei's essay on pacing frontier AI, 2. Satya Nadella's take on control in AI, 3. the Fields Medalists' report on misalignment in AI for mathematics, and 4. Anthropic's threat intelligence report have already triggered discussions around AI alignment, safety, risk, and control; on the other side, there is the rapid rise of forward-deployed engineering and inference engineering.

Last week I happened to participate in OpenAI's Astra hackathon at their NYC office and also attended Palantir's invite-only demo event. Across both events I could see one thing: people want to solve complex problems beyond the merely complicated, layered hyper-automation problems, which have started saturating or plummeting in terms of curiosity, learning curve, and also the human expertise and competence involved in them.

Be it AI's scale, applied engineering, or safety/risk evaluations, we have enough apparatus to understand, solve, and evaluate them with coding agents and human intervention. As we solve the low-hanging fruit, what emerges next are problems of safety, adoption, value creation, and long-term sustainability. These categories of problems are not linear or straightforward and can't be approached with a simple playbook; we need to understand the concept of complex systems and systems thinking to improve our problem-solving.

In the book Doughnut Economics, Kate Raworth argues for the need to understand economics as an organism rather than a mechanism. She explains why a statistical-mechanics/Newtonian way of approaching economics wouldn't help much and why we need to understand what a complex system is. Similarly, an Alignment Forum blog post emphasizes a "systems view" for pragmatic AI safety.

I have been exploring and learning about the field of complexity science as a data scientist since the early days of my career, through the influence of my first workplace, Mu Sigma. Since then, almost all of the problems I have happened to work on have been approached from the point of view of complex systems.

Let's see what a complex system is and why we need it across the range of real-world problems, including AI safety.

Here I'm just going to present what a complex system is and why it is needed. A complex system is made up of parts that interact with each other. So first, what is a complex system? Complex systems have multiple interconnected components. They interact with each other, and as a whole they form emergent behavior. When we want to analyze the individual parts, reductive analysis would not help us understand the behavior or solve the problems associated with it. Typically, people coming from a systems thinking background define elements like stocks, flows, and feedback. Feedback could be positive or negative. As a whole, these interconnected elements create a non-linear causal structure, and the effects are stochastic in nature. Non-linearity, interconnectedness, and emergent behavior are all characteristics of a complex system. Examples of complex systems include:

  • social networks
  • animal behavior
  • the internet
  • finance and financial markets
  • supply chains
  • biology, from cell behavior in the body to organs and disease

Even if you consider an urban ecosystem, you'll see that the individual elements are interconnected through some relationship or interaction, which exists as a flow of information, material, or energy. There exists some sort of delay or non-linearity, and as they keep interacting they form a new behavior, an emergent behavior, which may not be native to the individual components. Emergent behaviors arise when these interactions and feedbacks happen together.

Why is this so important? A typical linear, reductionist, or mechanism-based way of solving problems does not capture these interconnections, non-linearities, causal mechanisms, or feedbacks. Most of the time we keep things very localized to a specific component and always want to solve at that component level, but we miss understanding how the other aspects work together.

Even in consumer behavior, data scientists rely most of the time on the existing dataset, but they don't see things holistically or try to understand what surrounds it. This persists even in most modern scientific problems. Core research can ideally be directed toward a specific component, but when it comes to real-world phenomena, we more or less need to think from the complex systems point of view.

As I mentioned, it is not a one-solution-for-all-problems approach; it's about how we categorize problems and how thinking in terms of complex systems helps bring excellence or a better way to solve them.

For data scientists, to draw a distinction: if you just take market research or consumer lifecycle-related problems, we rely on whatever data exists in the database and try to use some linear models or machine learning models to predict something. In practice, data science is least bothered about either the causal structure or how customers' behavior forms and emerges when an intervention is given (maybe an advertisement or an offer), or what happens when new seasonality or something else is occurring.

I really like the classic starting point of going beyond correlation, but again, the question is how we can understand the interactions of various elements. When it comes to software engineering, we won't be able to build one holistic piece of software that helps with everything. Ideally, there is no issue in considering one specific complicated problem that needs different layers of solutions.

Think of a jet engine as a complicated solution, because it involves precise engineering of various components, but it doesn't really exhibit emergent behavior, since the feedback or interactions don't create new behavior at all. But if you take social media, or human–AI systems at an overall level, we can expect, over time, the responses to improve, or to have an effect on different communities' productivity, or humans to come up with new norms or new behaviors as they keep interacting with the AI, among many other things.

In the same way, when I was reading the blog post from the Alignment Forum, I could see that pragmatic AI safety is not just a single component. It has a lot of socio-technical factors that have an impact at various levels, be it research, policy, individuals, malicious actors, law, research adoption, and many other things. It involves a very different set of players, different feedbacks, incentives, and local mechanisms, and interactions that lead to the emergence of new behaviors, alignment issues, or safety-related problems.

A couple of days back, I tried to compare the diffusion of personal agents to the Indian myna in Australian suburbs, which started adapting and also creating issues for people and the ecosystem in Australia because of its emergent behavior and continuous interactions. If personal agents end up interacting with others, they might turn into something similar. Diffused AI systems, as they interact with humans and keep interacting with people from different backgrounds and with different self-interests, have a similar influence:

  • influence on social norms and emergent behavior among humans;
  • new outcomes in terms of well-being, safety, and many other things.

Let's take human–AI systems. On both sides there exists an exchange of conversations, preferences, and critiques or feedback, which in turn influences the memory or the .md files that all these personal agents keep. Since this one-on-one relationship doesn't have too many interconnected elements — just one agent and one human, a one-to-one mapping — it may seem simple. But I can still see how, at the individual level, the experience with these personal agents or AI systems creates productivity gains, or stress on well-being, or psychosis, and at a demographic or cultural level there always exist some emerging patterns that we could notice. Whether they turn out to be a problem or not is something one might need to keep looking for evidence and supporting scenarios for. But ideally, they have the potential to turn into annoying emergent behaviors or risky aspects as well, because, as I mentioned, we humans and AI systems together form a sort of complex system. There exist swarms of agents one can develop to achieve some goal. When these agents interact with each other, some sort of coordination can emerge, or cheating or lying can emerge as well — reward hacking is one such emergent behavior, I suspect. With all this having been said, the need to see or think from a systems or complexity point of view is really needed to address some of the critical problems and the blind spots we have from a typical approach.

Also, on the other side, in forward-deployed engineering, if you take Palantir's way of solving problems, it is predominantly a combination of having the right system (so that it helps you get data, as they mention) and building the digital twin of the business.

When they say "build the digital twin of the business," not just "have the systems of record of the business," it ideally means something different. If we just think of a typical way of solving problems, be it through automation, data science, or machine learning, it's always just to have the systems of record, clean things up, and then put some models, automations, or logic in place.

A typical Palantir ontology, however, connects different elements, like how people make decisions, how those flow back to the systems, and how they form an interconnectedness. With that, they bring in the operations layer: how one can see and capture all the possible elements through data, logic, and flows. The ontology is very similar to how typical complex systems are visualized through stocks, flows, delays, and feedback. In a similar way, one can see it here as well.

In my first job, we used to treat any problem space as a complex system and come up with individual components as hypotheses. With the help of data and analytics, we used to reduce the noise or get to the specific thing to focus on, to get started. Be it a healthcare real-world evidence problem or otherwise, instead of starting with the inductive bias of data, we started with the how, what, and why of whatever the target system was and defined it through semantic representations, assumptions, hypotheses, and questions. That led to how we validated and how we localized toward a specific segment.

If one could see AI engineering problems beyond typical automation, one could move between simple and complicated things. For example, a simple thing could be coming up with a very small agent, which can be built through Glean or some agent platform, which is very simple and straightforward. All one might need to do is connect some APIs and write the prompt. The end outcome is more or less about giving a sense of it. Sometimes it feels like a very engineered delusion: we are just going to build a chatbot and give it to the users, and they'll get answers, but do they get solutions? That's another question. In a complicated problem it's the same thing, but we want to serve a lot of people.

That doesn't account for these emergent behaviors at all; more or less, we just want to scale to a lot of people. We want to reduce the time to serve and dump more and more data into it. That exists at different levels of software engineering complicatedness, in terms of how many people and how much data we want to access and serve.

Take problems that involve too many entities forming these interconnections and feedback loops. If we just don't acknowledge them, if we just go ahead with "Okay, we have built a very good platform where you can connect all the data, and we have an agent, or you can just come up with your own API, and we can solve it, and we have a forward-deployed engineer," then fundamentally these companies are doing something wrong, or the problem-solving mindset is quite questionable here.

I see living inside the customer's problem as something Palantir used to define for their forward-deployed engineering. That setup of understanding the problem, owning the problem, and developing the required components, along with a substrate where the ontology can help connect all these elements, is what makes this problem-solving journey feasible. When I was working at Mu Sigma, we had a similar setup, but we had a very good framework to understand the problems very well. If one checks Mu Sigma's website, there is something called the Art of Problem Solving Systems, which helps in identifying all these critical elements and keeps questioning: do they make sense?

It's quite involved — a composition of Socratic and systematic thinking — and from there it moves to data-driven, or evidence-driven, problem solving. With that perspective, for any AI agents, AI safety, or critical high-stakes problems in insurance, healthcare, or even finance, I can see that human–AI systems or AI-based problem solving need this approach, rather than sophisticated or complicated systems integrations only. In reality, such integrations at times create illusions of solutions, but they don't really get to the problem-solving level.

There also exist a lot of companies or organizations that use the name "forward-deployed engineers," but ideally they don't get to the customers' problems in this much detail. They just ask what kind of systems the customer has and what kind of questions they want answered, and all they do is different levels of integrations. Think of a company that sells you forward-deployed engineers along with an NL-to-SQL product, or a company that sells you some document synthesis along with photo deduplication.

The question is, for these kinds of problems (be it analytics, CRM, document synthesis, or document automation), they ideally shouldn't need external billing or external partnerships just for implementation. The product shouldn't be this complicated or need someone in the name of a forward-deployed engineer, because in the AI era everyone is building AI-native solutions. All they are going to do is requirement collection and integrations. Ideally, they are not really visualizing what the core problem is — whether revenue is dipping, whether any root-cause analysis is needed, or maybe how to improve marketing performance.

Most of these platforms don't really understand the causality, interconnectedness, and stochasticity in these problems. They just directly jump in, audit the systems, connect everything, and tweak the prompts. I really don't think that is problem-solving, or that something like Palantir-level forward-deployed engineering is actually aimed at doing that. The name sounds fancy, but the ethos, pathos, and logos of such problem-solving are not at all being followed.

Ideally, be it AI engineering, safety, or even at the enterprise or organizational level, we might need to think from a complex systems point of view and then layer over that with data, automation engineering, and assistance engineering, rather than treating software integrations as problem-solving or expecting them to solve the problems. It may just create: "Okay, we have access to some agents, but do they really solve the problems?" That is something we need to ponder, because the question is: are we really solving them?

If you are making decisions about solving the problems you have, be it in cost savings, decision-making, or planning, you might need to really think about that. Even if I go back to what I learned through the book AI Snake Oil, most predictive AI projects failed or ended up as snake oil because they didn't really think through the problems. They just collected the data, built the model, and served it — but did they really go through the deep questioning of what we are really solving, what the population is, what the demographics are, whether there is any distributional shift, whether there are any other factors, and how the interconnections exist?

If you just reflect on the typical machine learning or predictive AI of the past, most of the time it was the same linear and reductionist approach. Now, with agents and superintelligent LLMs in place, I can see similar patterns: just building some wrappers and scalable systems. It's all fine for the software to be stable and always up and running, but the downstream solutions that really solve the problem and create value need more questioning, more systems thinking, more thinking than renaming and just dumping whatever you have in the name of context engineering or agent engineering.

References

  1. Dario Amodei, We Must Pace the Frontier, September 2026.
  2. Satya Nadella, post on X, 13 September 2026: "Any pursuit of superintelligence has to be grounded in the core principle that if the AI we build is not helping humanity and under human control, it's not worth pursuing."
  3. Math and AI, A Severe Misalignment of AI in Mathematics — a declaration endorsed by Fields Medalists, 2026.
  4. Anthropic, Detecting and countering misuse of AI: September 2026, September 2026.
  5. Kate Raworth, Doughnut Economics: Seven Ways to Think Like a 21st-Century Economist (2017).
  6. Dan Hendrycks and Thomas Woodside, Complex Systems for AI Safety (Pragmatic AI Safety #3), AI Alignment Forum, 24 May 2022.
  7. Arvind Narayanan and Sayash Kapoor, AI Snake Oil: What Artificial Intelligence Can Do, What It Can't, and How to Tell the Difference (2024).