How to Set Up RAGAS and Run Your First LLM Evaluation Test

What is RAGAS?

RAGAS stands for Retrieval-Augmented Generation Assessment. It is an open-source evaluation framework built specifically to measure how well a RAG system performs, the kind that combines vector search with an LLM to answer questions.

In plain terms, RAGAS helps you test how good your LLM-based system is at:

Retrieving the right documents
Generating accurate and relevant answers
Sticking to the facts

LLMs can sound completely confident and still be wrong. So evaluating their accuracy and reliability is critical, especially when they sit inside customer-facing, medical, legal, or internal tools. If you are new to RAG itself, start with what is RAG, then come back here to test one.

What you will learn

By the end of this guide, you will know how to:

Set up a Python project for LLM evaluation
Install RAGAS and its dependencies
Run your first test case using real inputs
Read and understand the evaluation scores

Step 1: Prerequisites

Before setting up RAGAS, get these two things ready on your computer.

1. Python 3. RAGAS runs on Python, so you need Python 3.8 or higher. To check whether it is already installed, open your terminal or command prompt and run:

python3 --version

Checking the installed Python version in the terminal

If you do not have it, download and install it from https://www.python.org/downloads/.

2. A code editor. You need an editor to write and run your Python scripts. I recommend PyCharm Community Edition, especially if you are new to Python. It is beginner-friendly and handles virtual environments well. Download it (the Community version is enough) from https://www.jetbrains.com/pycharm/download/.

Once both are ready, you can move on to setting up the project.

Step 2: Create your project in PyCharm

1. Create a new project. Launch PyCharm and click New Project. Then:

Set the project name to LlmEvaluation
Make sure New environment using Virtualenv is selected
Under Base interpreter, choose the same Python version you saw when you ran python3 --version
Click Create

Creating a new project with a Virtualenv environment in PyCharm

You will land in an empty project, ready for the libraries.

2. Install the required libraries. Go to PyCharm → Settings → Project: LlmEvaluation → Python Interpreter. Click the + (Add) button and search for each package below. Once all are added, click Apply, then OK.

Package	Why you need it
ragas	The main framework for evaluating LLM responses
langchain-openai	Lets us connect OpenAI to RAGAS using LangChain’s LLM wrapper
pytest	Helps us write and run test cases easily
pytest-asyncio	Required to run async test cases, which RAGAS depends on
requests	Used to send API requests to get responses and documents

The required packages installed in the PyCharm interpreter settings

That is your project and environment set up. Next, the first test.

Step 3: Write your first LLM evaluation test

Now let us write a simple test to check how well an answer uses its retrieved documents, using a metric called Context Precision. (If that metric is new to you, I explain it in context precision and context recall.)

Right-click your LlmEvaluation project folder in PyCharm and select New → Python File. Name it test_context_precision.py and paste in this code:

import os
import pytest
from langchain_openai import ChatOpenAI
from ragas import SingleTurnSample
from ragas.llms import LangchainLLMWrapper
from ragas.metrics import LLMContextPrecisionWithoutReference


@pytest.mark.asyncio
async def test_context_precision():
    os.environ["OPENAI_API_KEY"] = "replace your API key"
    # temperature=0 keeps the answer consistent; raise it if you want more elaboration
    llm = ChatOpenAI(model="gpt-3.5-turbo", temperature=0)
    langchain_llm = LangchainLLMWrapper(llm)
    context_precision = LLMContextPrecisionWithoutReference(llm=langchain_llm)

    sample = SingleTurnSample(
        user_input="What is the cost of living for a single?",
        response="Around S$2,660/month",
        retrieved_contexts=[
            "EP holders living alone spend around S$2,660/month, while S Pass holders spend about S$1,603/month",
            "Monthly expenses range between S$4,800 and S$5,500 for couples without kids. Monthly expenses range between S$6,500 and S$7,500 for couples with kids."
        ]
    )

    score = await context_precision.single_turn_ascore(sample)
    print(score)
    assert score > 0.8

Let me walk through what each part does.

The imports. os sets environment variables. pytest is the testing framework you run this with. ChatOpenAI lets us talk to OpenAI’s GPT model. SingleTurnSample represents one question-answer pair plus its context. LangchainLLMWrapper converts the LLM into a format RAGAS understands. LLMContextPrecisionWithoutReference is the metric we are testing.

The test function. The @pytest.mark.asyncio decorator and async def are there because RAGAS makes LLM calls asynchronously, so the test has to support that.

The API key. Replace "replace your API key" with your real OpenAI API key, or better, load it from a .env file. This is what lets the code connect to GPT.

The model. ChatOpenAI(model="gpt-3.5-turbo", temperature=0) sets up the GPT model. temperature=0 keeps the answer consistent and focused.

The wrapper. LangchainLLMWrapper(llm) wraps the LLM in a format RAGAS understands internally. LangChain is a Python library that makes it easier to work with LLMs like GPT. Here it acts as a bridge between you, the model, and RAGAS. Without this wrapper, RAGAS could not talk to the LLM directly, which is why wrapping the model is a required step.

The metric. LLMContextPrecisionWithoutReference(llm=langchain_llm) creates the metric object. Context precision answers: out of all the documents retrieved, how many were actually useful in answering the question?

The sample. SingleTurnSample defines the test input: user_input is the question, response is what the LLM answered, and retrieved_contexts are the documents that came back from the database. RAGAS then evaluates how well the answer matches that context.

One important note. For this guide, the user_input, response, and retrieved_contexts are hardcoded to keep things simple. In a real system, these should come dynamically from your RAG pipeline: the actual question a user asks, the actual output from your LLM, and the actual documents returned by your vector database, not values you type in by hand.

The score and assertion. score = await context_precision.single_turn_ascore(sample) runs the evaluation and prints a precision score between 0 and 1. A 1.0 means every retrieved document was relevant; a 0.0 means none of it helped. The final assert score > 0.8 says the test should only pass if at least 80% of the retrieved context was useful.

How to run the test

Open the terminal inside PyCharm and run:

pytest test_context_precision.py

You will see whether the test passed, along with the actual score:

============================= test session starts ==============================
collecting ... collected 1 item
test_context_precision.py::test_context_precision PASSED                                  [100%]0.9999999999
======================== 1 passed, 7 warnings in 2.08s =========================

Try a failing case

Now let us simulate a failure by feeding in irrelevant context. Replace the previous sample = SingleTurnSample(...) block with this:

sample = SingleTurnSample(
    user_input="What is the cost of living for a single in Singapore?",
    response="Couldn't find answer",
    retrieved_contexts=[
        "Singapore is known for its cultural diversity and vibrant nightlife.",
        "Tourist arrivals in Singapore dropped significantly in 2020 due to the pandemic."
    ]
)

This time the test fails, and the score drops to zero:

============================= test session starts ==============================
collecting ... collected 1 item
test_context_precision.py::test_context_precision FAILED                                  [100%]0.0
Test1.py:10 (test_context_precision)
0.0 != 0.8
Expected :0.8
Actual   :0.0

That is exactly how RAGAS catches bad retrieval.

What you have done

You have set up RAGAS, written your first test case, and learned how to measure the quality of an LLM’s answer using context precision. We used hardcoded data here to keep it simple, but you now have the foundation to test real RAG systems.

In the next article, I will show how to run dynamic evaluations using real API responses, connecting to an actual RAG system and scoring its live answers instead of hardcoded samples.

That’s it for today, guys. Thank You for Reading! I hope you found this article informative and useful.

If you think it could benefit others, please share it on your social media networks with friends and family who might also appreciate it.