Accuracy is often critical for LLM applications, especially in cases such as API calling or summarisation of financial reports. Fortunately, there are ways to enhance precision. The best practices to improve accuracy include the following steps:
- You can start simply with prompt engineering techniques — adding more detailed instructions, using few-shot prompting, or asking the model to think step-by-step.
- If accuracy is still insufficient, you can incorporate a self-reflection step, for example, to return errors from the API calls and ask the LLM to correct mistakes.
- The next option is to provide the most relevant context to the LLM using RAG (Retrieval-Augmented Generation) to boost precision further.
We’ve explored this approach in my previous TDS article, “From Prototype to Production: Enhancing LLM Accuracy”. In that project, we built an SQL Agent and went from 0% valid SQL queries to 70% accuracy. However, there are limits to what we can achieve with prompt. To break through this barrier and reach the next frontier of accuracy, we need to adopt more advanced techniques.
The most promising option is fine-tuning. With fine-tuning, we can move from relying solely on information in prompts to embedding additional information directly into the model’s weights.
Let’s start by understanding what fine-tuning is. Fine-tuning is the process of refining pre-trained models by training them on smaller, task-specific datasets to enhance their performance in particular applications. Basic models are initially trained on vast amounts of data, which allows them to develop a broad understanding of language. Fine-tuning, however, tailors these models to specialized tasks, transforming them from general-purpose systems into highly targeted tools. For example, instruction fine-tuning taught GPT-2 to chat and follow instructions, and that’s how ChatGPT emerged.
Basic LLMs are initially trained to predict the next token based on vast text corpora. Fine-tuning typically adopts a supervised approach, where the model is presented with specific questions and corresponding answers, allowing it to adjust its weights to improve accuracy.
Historically, fine-tuning required updating all model weights, a method known as full fine-tuning. This process was computationally expensive since it required storing all the model weights, states, gradients and forward activations in memory. To address these challenges, parameter-efficient fine-tuning techniques were introduced. PEFT methods update only the small set of the model parameters while keeping the rest frozen. Among these methods, one of the most widely adopted is LoRA (Low-Rank Adaptation), which significantly reduces the computational cost without compromising performance.
Pros & cons
Before considering fine-tuning, it’s essential to weigh its advantages and limitations.
Advantages:
- Fine-tuning enables the model to learn and retain significantly more information than can be provided through prompts alone.
- It usually gives higher accuracy, often exceeding 90%.
- During inference, it can reduce costs by enabling the use of smaller, task-specific models instead of larger, general-purpose ones.
- Fine-tuned small models can often be deployed on-premises, eliminating reliance on cloud providers such as OpenAI or Anthropic. This approach reduces costs, enhances privacy, and minimizes dependency on external infrastructure.
Disadvantages:
- Fine-tuning requires upfront investments for model training and data preparation.
- It requires specific technical knowledge and may involve a steep learning curve.
- The quality of results depends heavily on the availability of high-quality training data.
Since this project is focused on gaining knowledge, we will proceed with fine-tuning. However, in real-world scenarios, it’s important to evaluate whether the benefits of fine-tuning justify all the associated costs and efforts.
Execution
The next step is to plan how we will approach fine-tuning. After listening to the “Improving Accuracy of LLM Applications” course, I’ve decided to try the Lamini platform for the following reasons:
- It offers a simple one-line API call to fine-tune the model. It’s especially convenient since we’re just starting to learn a new technique.
- Although it’s not free and can be quite expensive for toy projects (at $1 per tuning step), they offer free credits upon registration, which are sufficient for initial testing.
- Lamini has implemented a new approach, Lamini Memory Tuning, which promises zero loss of factual accuracy while preserving general capabilities. This is a significant claim, and it’s worth testing out. We will discuss this approach in more detail shortly.
Of course, there are lots of other fine-tuning options you can consider:
- The Llama documentation provides numerous recipes for fine-tuning, which can be executed on a cloud server or even locally for smaller models.
- There are many step-by-step guides available online, including the tutorial on how to fine-tune Llama on Kaggle from DataCamp.
- You can fine-tune not only open-sourced models. OpenAI also offers the capability to fine-tune their models.
Lamini Memory Tuning
As I mentioned earlier, Lamini released a new approach to fine-tuning, and I believe it’s worth discussing it in more detail.
Lamini introduced the Mixture of Memory Experts (MoME) approach, which enables LLMs to learn a vast amount of factual information with almost zero loss, all while maintaining generalization capabilities and requiring a feasible amount of computational resources.
To achieve this, Lamini extended a pre-trained LLM by adding a large number (on the order of 1 million) of LoRA adapters along with a cross-attention layer. Each LoRA adapter is a memory expert, functioning as a type of memory for the model. These memory experts specialize in different aspects, ensuring that the model retains faithful and accurate information from the data it was tuned on. Inspired by information retrieval, these million memory experts are equivalent to indices from which the model intelligently retrieves and routes.
At inference time, the model retrieves a subset of the most relevant experts at each layer and merges back into the base model to generate a response to the user query.
Lamini Memory Tuning is said to be capable of achieving 95% accuracy. The key difference from traditional instruction fine-tuning is that instead of optimizing for average error across all tasks, this approach focuses on achieving zero error for the facts the model is specifically trained to remember.
So, this approach allows an LLM to preserve its ability to generalize with average error on everything else while recalling the important facts nearly perfectly.
For further details, you can refer to the research paper “Banishing LLM Hallucinations Requires Rethinking Generalization” by Li et al. (2024)
Lamini Memory Tuning holds great promise — let’s see if it delivers on its potential in practice.
As always, let’s begin by setting everything up. As we discussed, we’ll be using Lamini to fine-tune Llama, so the first step is to install the Lamini package.
pip install lamini
Additionally, we need to set up the Lamini API Key on their website and specify it as an environment variable.
export LAMINI_API_KEY="<YOUR-LAMINI-API-KEY>"
As I mentioned above, we will be improving the SQL Agent, so we need a database. For this example, we’ll continue using ClickHouse, but feel free to choose any database that suits your needs. You can find more details on the ClickHouse setup and the database schema in the previous article.
To fine-tune an LLM, we first need a dataset — in our case, a set of pairs of questions and answers (SQL queries). The task of putting together a dataset might seem daunting, but luckily, we can leverage LLMs to do it.
The key factors to consider while preparing the dataset:
- The quality of the data is crucial, as we will ask the model to remember these facts.
- Diversity in the examples is important so that a model can learn how to handle different cases.
- It’s preferable to use real data rather than synthetically generated data since it better represents real-life questions.
- The usual minimum size for a fine-tuning dataset is around 1,000 examples, but the more high-quality data, the better.
Generating examples
All the information required to create question-and-answer pairs is present in the database schema, so it will be a feasible task for an LLM to generate examples. Additionally, I have a representative set of Q&A pairs that I used for RAG approach, which we can present to the LLM as examples of valid queries (using the few-shot prompting technique). Let’s load the RAG dataset.
# loading a set of examples
with open('rag_set.json', 'r') as f:
rag_set = json.loads(f.read())rag_set_df = pd.DataFrame(rag_set)
rag_set_df['qa_fmt'] = list(map(
lambda x, y: "question: %s, sql_query: %s" % (x, y),
rag_set_df.question,
rag_set_df.sql_query
))
The idea is to iteratively provide the LLM with the schema information and a set of random examples (to ensure diversity in the questions) and ask it to generate a new, similar, but different Q&A pair.
Let’s create a system prompt that includes all the necessary details about the database schema.
generate_dataset_system_prompt = '''
You are a senior data analyst with more than 10 years of experience writing complex SQL queries.
There are two tables in the database you're working with with the following schemas. Table: ecommerce.users
Description: customers of the online shop
Fields:
- user_id (integer) - unique identifier of customer, for example, 1000004 or 3000004
- country (string) - country of residence, for example, "Netherlands" or "United Kingdom"
- is_active (integer) - 1 if customer is still active and 0 otherwise
- age (integer) - customer age in full years, for example, 31 or 72
Table: ecommerce.sessions
Description: sessions for online shop
Fields:
- user_id (integer) - unique identifier of customer, for example, 1000004 or 3000004
- session_id (integer) - unique identifier of session, for example, 106 or 1023
- action_date (date) - session start date, for example, "2021-01-03" or "2024-12-02"
- session_duration (integer) - duration of session in seconds, for example, 125 or 49
- os (string) - operation system that customer used, for example, "Windows" or "Android"
- browser (string) - browser that customer used, for example, "Chrome" or "Safari"
- is_fraud (integer) - 1 if session is marked as fraud and 0 otherwise
- revenue (float) - income in USD (the sum of purchased items), for example, 0.0 or 1506.7
Write a query in ClickHouse SQL to answer the following question.
Add "format TabSeparatedWithNames" at the end of the query to get data from ClickHouse database in the right format.
'''
The next step is to create a template for the user query.
generate_dataset_qa_tmpl = '''
Considering the following examples, please, write question
and SQL query to answer it, that is similar but different to provided below.Examples of questions and SQL queries to answer them:
examples
'''
Since we need a high-quality dataset, I prefer using a more advanced model — GPT-4o— rather than Llama. As usual, I’ll initialize the model and create a dummy tool for structured output.
from langchain_core.tools import tool@tool
def generate_question_and_answer(comments: str, question: str, sql_query: str) -> str:
"""Returns the new question and SQL query
Args:
comments (str): 1-2 sentences about the new question and answer pair,
question (str): new question
sql_query (str): SQL query in ClickHouse syntax to answer the question
"""
pass
from langchain_openai import ChatOpenAI
generate_qa_llm = ChatOpenAI(model="gpt-4o", temperature = 0.5)\
.bind_tools([generate_question_and_answer])
Now, let’s combine everything into a function that will generate a Q&A pair and create a set of examples.
# helper function to combine system + user prompts
def get_openai_prompt(question, system):
messages = [
("system", system),
("human", question)
]
return messagesdef generate_qa():
# selecting 3 random examples
sample_set_df = rag_set_df.sample(3)
examples = '\n\n'.join(sample_set_df.qa_fmt.values)
# constructing prompt
prompt = get_openai_prompt(
generate_dataset_qa_tmpl.format(examples = examples),
generate_dataset_system_prompt)
# calling LLM
qa_res = generate_qa_llm.invoke(prompt)
try:
rec = qa_res.tool_calls[0]['args']
rec['examples'] = examples
return rec
except:
pass
# executing function
qa_tmp = []
for i in tqdm.tqdm(range(2000)):
qa_tmp.append(generate_qa())
new_qa_df = pd.DataFrame(qa_tmp)
I generated 2,000 examples, but in reality, I used a much smaller dataset for this toy project. Therefore, I recommend limiting the number of examples to 200–300.
Cleaning the dataset
As we know, “garbage in, garbage out”, so an essential step before fine-tuning is cleaning the data generated by the LLM.
The first — and most obvious — check is to ensure that each SQL query is valid.
def is_valid_output(s):
if s.startswith('Database returned the following error:'):
return 'error'
if len(s.strip().split('\n')) >= 1000:
return 'too many rows'
return 'ok'new_qa_df['output'] = new_qa_df.sql_query.map(get_clickhouse_data)
new_qa_df['is_valid_output'] = new_qa_df.output.map(is_valid_output)
There are no invalid SQL queries, but some questions return over 1,000 rows.
Although these cases are valid, we’re focusing on an OLAP scenario with aggregated stats, so I’ve retained only queries that return 100 or fewer rows.
new_qa_df['output_rows'] = new_qa_df.output.map(
lambda x: len(x.strip().split('\n')))filt_new_qa_df = new_qa_df[new_qa_df.output_rows <= 100]
I also eliminated cases with empty output — queries that return no rows or only the header.
filt_new_qa_df = filt_new_qa_df[filt_new_qa_df.output_rows > 1]
Another important check is for duplicate questions. The same question with different answers could confuse the model, as it won’t be able to tune to both solutions simultaneously. And in fact, we have such cases.
filt_new_qa_df = filt_new_qa_df[['question', 'sql_query']].drop_duplicates()
filt_new_qa_df['question'].value_counts().head(10)
To resolve these duplicates, I’ve kept only one answer for each question.
filt_new_qa_df = filt_new_qa_df.drop_duplicates('question')
Although I generated around 2,000 examples, I’ve decided to use a smaller dataset of 200 question-and-answer pairs. Fine-tuning with a larger dataset would require more tuning steps and be more expensive.
sample_dataset_df = pd.read_csv('small_sample_for_finetuning.csv', sep = '\t')
You can find the final training dataset on GitHub.
Now that our training dataset is ready, we can move on to the most exciting part — fine-tuning.
The first iteration
The next step is to generate the sets of requests and responses for the LLM that we will use to fine-tune the model.
Since we’ll be working with the Llama model, let’s create a helper function to construct a prompt for it.
def get_llama_prompt(user_message, system_message=""):
system_prompt = ""
if system_message != "":
system_prompt = (
f"<|start_header_id|>system<|end_header_id|>\n\nsystem_message"
f"<|eot_id|>"
)
prompt = (f"<|begin_of_text|>system_prompt"
f"<|start_header_id|>user<|end_header_id|>\n\n"
f"user_message"
f"<|eot_id|>"
f"<|start_header_id|>assistant<|end_header_id|>\n\n"
)
return prompt
For requests, we will use the following system prompt, which includes all the necessary information about the data schema.
generate_query_system_prompt = '''
You are a senior data analyst with more than 10 years of experience writing complex SQL queries.
There are two tables in the database you're working with with the following schemas. Table: ecommerce.users
Description: customers of the online shop
Fields:
- user_id (integer) - unique identifier of customer, for example, 1000004 or 3000004
- country (string) - country of residence, for example, "Netherlands" or "United Kingdom"
- is_active (integer) - 1 if customer is still active and 0 otherwise
- age (integer) - customer age in full years, for example, 31 or 72
Table: ecommerce.sessions
Description: sessions of usage the online shop
Fields:
- user_id (integer) - unique identifier of customer, for example, 1000004 or 3000004
- session_id (integer) - unique identifier of session, for example, 106 or 1023
- action_date (date) - session start date, for example, "2021-01-03" or "2024-12-02"
- session_duration (integer) - duration of session in seconds, for example, 125 or 49
- os (string) - operation system that customer used, for example, "Windows" or "Android"
- browser (string) - browser that customer used, for example, "Chrome" or "Safari"
- is_fraud (integer) - 1 if session is marked as fraud and 0 otherwise
- revenue (float) - income in USD (the sum of purchased items), for example, 0.0 or 1506.7
Write a query in ClickHouse SQL to answer the following question.
Add "format TabSeparatedWithNames" at the end of the query to get data from ClickHouse database in the right format.
Answer questions following the instructions and providing all the needed information and sharing your reasoning.
'''
Let’s create the responses in the format suitable for Lamini fine-tuning. We need to prepare a list of dictionaries with input and output keys.
formatted_responses = []for rec in sample_dataset_df.to_dict('records'):
formatted_responses.append(
'input': get_llama_prompt(rec['question'],
generate_query_system_prompt),
'output': rec['sql_query']
)
Now, we are fully prepared for fine-tuning. We just need to select a model and initiate the process. We will be fine-tuning the Llama 3.1 8B model.
from lamini import Lamini
llm = Lamini(model_name="meta-llama/Meta-Llama-3.1-8B-Instruct")finetune_args =
"max_steps": 50,
"learning_rate": 0.0001
llm.train(
data_or_dataset_id=formatted_responses,
finetune_args=finetune_args,
)
We can specify several hyperparameters, and you can find all the details in the Lamini documentation. For now, I’ve passed only the most essential ones to the function:
max_steps: This determines the number of tuning steps. The documentation recommends using 50 steps for experimentation to get initial results without spending too much money.learning_rate: This parameter determines the step size of each iteration while moving toward a minimum of a loss function (Wikipedia). The default is 0.0009, but based on the guidance, I’ve decided to use a smaller value.
Now, we just need to wait for 10–15 minutes while the model trains, and then we can test it.
finetuned_llm = Lamini(model_name='<model_id>')
# you can find Model ID in the Lamini interfacequestion = '''How many customers made purchase in December 2024?'''
prompt = get_llama_prompt(question, generate_query_system_prompt)
finetuned_llm.generate(prompt, max_new_tokens=200)
# select uniqExact(s.user_id) as customers
# from ecommerce.sessions s join ecommerce.users u
# on s.user_id = u.user_id
# where (toStartOfMonth(action_date) = '2024-12-01') and (revenue > 0)
# format TabSeparatedWithNames
It’s worth noting that we’re using Lamini for inference as well and will have to pay for it. You can find up-to-date information about the costs here.
At first glance, the result looks promising, but we need a more robust accuracy evaluation to confirm it.
Additionally, it’s worth noting that since we’ve fine-tuned the model for our specific task, it now consistently returns SQL queries, meaning we may no longer need to use tool calls for structured output.
Evaluating the quality
We’ve discussed LLM accuracy evaluation in detail in my previous article, so here I’ll provide a brief recap.
We use a golden set of question-and-answer pairs to evaluate the model’s quality. Since this is a toy example, I’ve limited the set to just 10 pairs, which you can review on GitHub.
The evaluation process consists of two parts:
- SQL Query Validity: First, we check that the SQL query is valid, meaning ClickHouse doesn’t return errors during execution.
- Query Correctness: Next, we ensure that the generated query is correct. We compare the outputs of the generated and true queries using LLMs to verify that they provide semantically identical results.
The initial results are far from ideal, but they are significantly better than the base Llama model (which produced zero valid SQL queries). Here’s what we found:
- ClickHouse returned errors for two queries.
- Three queries were executed, but the results were incorrect.
- Five queries were correct.
No surprises — there’s no silver bullet, and it’s always an iterative process. Let’s investigate what went wrong.
Diving into the errors
The approach is straightforward. Let’s examine the errors one by one to understand why we got these results and how we can fix them. We’ll start with the first unsuccessful example.