Sometimes, your “experiment” will fail, then you slightly pivot your work, and this other experiment succeeded much better.
That’s precisely why, before designing our endgame solution, we must start simple and hedge our risks.
- Define a “budget” or timeframe. Let’s see what we can do in X weeks and then decide how or if to continue. Usually, 2–4 weeks to understand basic PoC will be sufficient. If it looks promising — continue investing resources to improve it.
- Experiment—Whether you choose a bottom-up or top-down approach for your experimentation phase, your goal is to maximize the result succession rate. By the end of the first experimentation iteration, you should have some PoC (that stakeholders can play with) and a baseline you achieved.
- Retrospective — By the end of our research phase, we can understand the feasibility, limitations, and cost of building such an app. This helps us decide whether to productionize it and how to design the final product and its UX.
- Productization — Develop a production-ready version of your project and integrate it with the rest of your solution by following standard SWE best practices and implementing a feedback and data collection mechanism.
To implement the experiment-oriented process well, we must make an informed decision on approaching and constructing these experiments:
Starting Lean: The Bottom-Up Approach
While many early adopters quickly jump into” State-Of-The-Art” multichain agentic systems with full-fledged Langchain or something similar, I found “The Bottom-Up approach” often yields better results.
Start lean, very lean, embracing the “one prompt to rule them all” philosophy. Although this strategy might seem unconventional and will likely produce bad results at first, it establishes a baseline for your system.
From there, continuously iterate and refine your prompts, employing prompt engineering techniques to optimize outcomes. As you identify weaknesses in your lean solution, split the process by adding branches to address those shortcomings.
While designing each “leaf” of my LLM workflow graph, or LLM-native architecture, I follow The Magic Triangle³ to determine where and when to cut the branches, split them, or thicken the roots (by using prompt engineering techniques) and squeeze more of the lemon.
For example, to implement “Native language SQL querying” with the bottom-up approach, we’ll start by naively sending the schemas to the LLM and ask it to generate a query.
Usually, this does not contradict the “top-down approach” but serves as another step before it. This allows us to show quick wins and attract more project investment.
The Big Picture Upfront: The Top-Down Strategy
“We know that LLM workflow is not easy, and to achieve our goal, we’ll probably end up with some workflow or LLM-native architecture.”
The Top-Down approach recognizes it and starts by designing the LLM-native architecture from day one and implementing its different steps/chains from the beginning.
This way, you can test your workflow architecture as a whole and squeeze the whole lemon instead of refining each leaf separately.
For example, to implement “Native language SQL querying” with the top-down approach, we’ll start designing the architecture before even starting to code and then jump to the full implementation:
Finding the Right Balance
When you start experimenting with LLMs, you’ll probably start at one of the extremes (overcomplicated top-down or super simple one-shot). In reality, there’s no such a winner.
Ideally — you’ll define a good SoP¹ and model an expert before coding and experimenting with the model. In reality, modeling is very hard; sometimes, you may not have access to such an expert.
I found it challenging to land on a good architecture/SoP¹ at the first shot, so it’s worth experimenting lightly before jumping to the big guns. However, it doesn’t mean that everything has to be too lean. If you already have a prior understanding that something MUST be broken into smaller pieces — do that.
In any case, you should leverage The Magic Triangle³ paradigm and model the manual process correctly while designing your solution.