Skip to content
Snippets Groups Projects
Commit 194bc3a2 authored by Embruch, Gerd's avatar Embruch, Gerd
Browse files

Initial commit

parents
Branches
Tags
No related merge requests found
Show whitespace changes
Inline Side-by-side
.env.template 0 → 100644
+ 22
− 0
View file @ 194bc3a2
# WHICH PORT SHOULD THE WEBSERVICE LISTEN ON?
PHOENIX_PORT=6006
# API KEY TO OPENAI / CHATGPT API
OPENAI_API_KEY=sk-hj8dfs8f9dsfsdgibberish
#CHUNK SIZE OF DOMAIN FILES
DOCUMENTS_CHUNK_SIZE=512
#HOW MANY QUESTIONS PER CHUNK SHALL BE GENERATED?
QUESTIONS_PER_CHUNK=2
# BASEPATH OF DOMAIN FILES
DOMAIN_BASEPATH=https://rodderberg.zbh.uni-hamburg.de/wiki/data/pages/
# FILES TO DOWNLOAD WITHOUT BASEPATH
DOMAIN_DOCS=amd/tutorials/programming/ide_setup.txt,amd/teaching/lessons_learned_lehre.txt
# DOWNLOAD FOLDER
DOMAIN_DOWNLOAD_FOLDER=./downloads
.gitignore 0 → 100644
+ 4
− 0
View file @ 194bc3a2
.env
.venv
.ipynb_checkpoints
downloads
README.md 0 → 100644
+ 25
− 0
View file @ 194bc3a2
# Prerequisits
- python3 installed
- [openAI API Key](https://auth.openai.com/)
# Install
```
python3 -m venv .venv
source .venv/bin/activate
pip install-r requirements.txt
cp ./.env.template ./.env
```
# Configure
populate the `.env` file with proper information
# Start
```
python3 evaluateRAG.py
```
acccess webservice via browser, i.e.
`firefox http:<HOSTNAME>:<PORT>`
# Sources
- [YT: RAG Time! Evaluate RAG with LLM Evals and Benchmarking](https://www.youtube.com/watch?v=LrMguHcbpO8)
- [Phoenix Docs](https://docs.arize.com/phoenix)
evaluateRAG.ipynb 0 → 100755
+ 1166
− 0
View file @ 194bc3a2
%% Cell type:markdown id: tags:
%% Cell type:markdown id: tags:
<center>
<p style="text-align:center">
<img alt="phoenix logo" src="https://storage.googleapis.com/arize-phoenix-assets/assets/phoenix-logo-light.svg" width="200"/>
<br>
<a href="https://docs.arize.com/phoenix/">Docs</a>
|
<a href="https://github.com/Arize-ai/phoenix">GitHub</a>
|
<a href="https://join.slack.com/t/arize-ai/shared_invite/zt-1px8dcmlf-fmThhDFD_V_48oU7ALan4Q">Community</a>
</p>
</center>
<h1 align="center">Evaluate RAG with LLM Evals</h1>
In this tutorial we will look into building a RAG pipeline and evaluating it with Phoenix Evals.
It has the the following sections:
1. Understanding Retrieval Augmented Generation (RAG).
2. Building RAG (with the help of a framework such as LlamaIndex).
3. Evaluating RAG with Phoenix Evals.
%% Cell type:markdown id: tags:
## Retrieval Augmented Generation (RAG)
LLMs are trained on vast datasets, but these will not include your specific data (things like company knowledge bases and documentation). Retrieval-Augmented Generation (RAG) addresses this by dynamically incorporating your data as context during the generation process. This is done not by altering the training data of the LLMs but by allowing the model to access and utilize your data in real-time to provide more tailored and contextually relevant responses.
In RAG, your data is loaded and prepared for queries. This process is called indexing. User queries act on this index, which filters your data down to the most relevant context. This context and your query then are sent to the LLM along with a prompt, and the LLM provides a response.
RAG is a critical component for building applications such a chatbots or agents and you will want to know RAG techniques on how to get data into your application.
<img src="https://storage.googleapis.com/arize-phoenix-assets/assets/images/RAG_Pipeline.png">
%% Cell type:markdown id: tags:
## Stages within RAG
There are five key stages within RAG, which will in turn be a part of any larger RAG application.
- **Loading**: This refers to getting your data from where it lives - whether it's text files, PDFs, another website, a database or an API - into your pipeline.
- **Indexing**: This means creating a data structure that allows for querying the data. For LLMs this nearly always means creating vector embeddings, numerical representations of the meaning of your data, as well as numerous other metadata strategies to make it easy to accurately find contextually relevant data.
- **Storing**: Once your data is indexed, you will want to store your index, along with any other metadata, to avoid the need to re-index it.
- **Querying**: For any given indexing strategy there are many ways you can utilize LLMs and data structures to query, including sub-queries, multi-step queries, and hybrid strategies.
- **Evaluation**: A critical step in any pipeline is checking how effective it is relative to other strategies, or when you make changes. Evaluation provides objective measures on how accurate, faithful, and fast your responses to queries are.
%% Cell type:markdown id: tags:
## Build a RAG system
Now that we have understood the stages of RAG, let's build a pipeline. We will use [LlamaIndex](https://www.llamaindex.ai/) for RAG and [Phoenix Evals](https://docs.arize.com/phoenix/llm-evals/llm-evals) for evaluation.
%% Cell type:code id: tags:
``` python
pip install -qq "arize-phoenix[evals]" "llama-index>=0.10.3" "openinference-instrumentation-llama-index>=1.0.0" "llama-index-callbacks-arize-phoenix>=0.1.2" "llama-index-llms-openai" "openai>=1" gcsfs nest_asyncio
```
%% Cell type:code id: tags:
``` python
# The nest_asyncio module enables the nesting of asynchronous functions within an already running async loop.
# This is necessary because Jupyter notebooks inherently operate in an asynchronous loop.
# By applying nest_asyncio, we can run additional async functions within this existing loop without conflicts.
import nest_asyncio
nest_asyncio.apply()
import os
from getpass import getpass
import pandas as pd
import phoenix as px
from llama_index.core import SimpleDirectoryReader, VectorStoreIndex, set_global_handler
from llama_index.core.node_parser import SimpleNodeParser
from llama_index.llms.openai import OpenAI
```
%% Cell type:markdown id: tags:
During this tutorial, we will capture all the data we need to evaluate our RAG pipeline using Phoenix Tracing. To enable this, simply start the phoenix application and instrument LlamaIndex.
%% Cell type:code id: tags:
``` python
session = px.launch_app()
```
%% Cell type:code id: tags:
``` python
set_global_handler("arize_phoenix")
```
%% Cell type:markdown id: tags:
For this tutorial we will be using OpenAI for creating synthetic data as well as for evaluation.
%% Cell type:code id: tags:
``` python
if not (openai_api_key := os.getenv("OPENAI_API_KEY")):
openai_api_key = getpass("🔑 Enter your OpenAI API key: ")
os.environ["OPENAI_API_KEY"] = openai_api_key
```
%% Cell type:markdown id: tags:
### Load Data and Build an Index
%% Cell type:markdown id: tags:
Let's use an [essay by Paul Graham](https://www.paulgraham.com/worked.html) to build our RAG pipeline.
%% Cell type:code id: tags:
``` python
import tempfile
from urllib.request import urlretrieve
with tempfile.NamedTemporaryFile() as tf:
urlretrieve(
"https://raw.githubusercontent.com/Arize-ai/phoenix-assets/main/data/paul_graham/paul_graham_essay.txt",
tf.name,
)
documents = SimpleDirectoryReader(input_files=[tf.name]).load_data()
```
%% Cell type:code id: tags:
``` python
# Define an LLM
llm = OpenAI(model="gpt-4")
# Build index with a chunk_size of 512
node_parser = SimpleNodeParser.from_defaults(chunk_size=512)
nodes = node_parser.get_nodes_from_documents(documents)
vector_index = VectorStoreIndex(nodes)
```
%% Cell type:markdown id: tags:
Build a QueryEngine and start querying.
%% Cell type:code id: tags:
``` python
query_engine = vector_index.as_query_engine()
```
%% Cell type:code id: tags:
``` python
response_vector = query_engine.query("What did the author do growing up?")
```
%% Cell type:markdown id: tags:
Check the response that you get from the query.
%% Cell type:code id: tags:
``` python
response_vector.response
```
%% Cell type:markdown id: tags:
By default LlamaIndex retrieves two similar nodes/ chunks. You can modify that in `vector_index.as_query_engine(similarity_top_k=k)`.
Let's check the text in each of these retrieved nodes.
%% Cell type:code id: tags:
``` python
# First retrieved node
response_vector.source_nodes[0].get_text()
```
%% Cell type:code id: tags:
``` python
# Second retrieved node
response_vector.source_nodes[1].get_text()
```
%% Cell type:markdown id: tags:
Remember that we are using Phoenix Tracing to capture all the data we need to evaluate our RAG pipeline. You can view the traces in the phoenix application.
%% Cell type:code id: tags:
``` python
print("phoenix URL", px.active_session().url)
```
%% Cell type:markdown id: tags:
We can access the traces by directly pulling the spans from the phoenix session.
%% Cell type:code id: tags:
``` python
spans_df = px.Client().get_spans_dataframe()
```
%% Cell type:code id: tags:
``` python
spans_df[["name", "span_kind", "attributes.input.value", "attributes.retrieval.documents"]].head()
```
%% Cell type:markdown id: tags:
Note that the traces have captured the documents that were retrieved by the query engine. This is nice because it means we can introspect the documents without having to keep track of them ourselves.
%% Cell type:code id: tags:
``` python
spans_with_docs_df = spans_df[spans_df["attributes.retrieval.documents"].notnull()]
```
%% Cell type:code id: tags:
``` python
spans_with_docs_df[["attributes.input.value", "attributes.retrieval.documents"]].head()
```
%% Cell type:markdown id: tags:
We have built a RAG pipeline and also have instrumented it using Phoenix Tracing. We now need to evaluate it's performance. We can assess our RAG system/query engine using Phoenix's LLM Evals. Let's examine how to leverage these tools to quantify the quality of our retrieval-augmented generation system.
%% Cell type:markdown id: tags:
## Evaluation
Evaluation should serve as the primary metric for assessing your RAG application. It determines whether the pipeline will produce accurate responses based on the data sources and range of queries.
While it's beneficial to examine individual queries and responses, this approach is impractical as the volume of edge-cases and failures increases. Instead, it's more effective to establish a suite of metrics and automated evaluations. These tools can provide insights into overall system performance and can identify specific areas that may require scrutiny.
In a RAG system, evaluation focuses on two critical aspects:
- **Retrieval Evaluation**: To assess the accuracy and relevance of the documents that were retrieved
- **Response Evaluation**: Measure the appropriateness of the response generated by the system when the context was provided.
%% Cell type:markdown id: tags:
### Generate Question Context Pairs
For the evaluation of a RAG system, it's essential to have queries that can fetch the correct context and subsequently generate an appropriate response.
For this tutorial, let's use Phoenix's `llm_generate` to help us create the question-context pairs.
%% Cell type:markdown id: tags:
First, let's create a dataframe of all the document chunks that we have indexed.
%% Cell type:code id: tags:
``` python
# Let's construct a dataframe of just the documents that are in our index
document_chunks_df = pd.DataFrame({"text": [node.get_text() for node in nodes]})
document_chunks_df.head()
```
%% Cell type:markdown id: tags:
Now that we have the document chunks, let's prompt an LLM to generate us 3 questions per chunk. Note that you could manually solicit questions from your team or customers, but this is a quick and easy way to generate a large number of questions.
%% Cell type:code id: tags:
``` python
generate_questions_template = """\
Context information is below.
---------------------
{text}
---------------------
Given the context information and not prior knowledge.
generate only questions based on the below query.
You are a Teacher/ Professor. Your task is to setup \
3 questions for an upcoming \
quiz/examination. The questions should be diverse in nature \
across the document. Restrict the questions to the \
context information provided."
Output the questions in JSON format with the keys question_1, question_2, question_3.
"""
```
%% Cell type:code id: tags:
``` python
import json
from phoenix.evals import OpenAIModel, llm_generate
def output_parser(response: str, index: int):
try:
return json.loads(response)
except json.JSONDecodeError as e:
return {"__error__": str(e)}
questions_df = llm_generate(
dataframe=document_chunks_df,
template=generate_questions_template,
model=OpenAIModel(
model_name="gpt-3.5-turbo",
),
output_parser=output_parser,
concurrency=20,
)
```
%% Cell type:code id: tags:
``` python
questions_df.head()
```
%% Cell type:code id: tags:
``` python
# Construct a dataframe of the questions and the document chunks
questions_with_document_chunk_df = pd.concat([questions_df, document_chunks_df], axis=1)
questions_with_document_chunk_df = questions_with_document_chunk_df.melt(
id_vars=["text"], value_name="question"
).drop("variable", axis=1)
# If the above step was interrupted, there might be questions missing. Let's run this to clean up the dataframe.
questions_with_document_chunk_df = questions_with_document_chunk_df[
questions_with_document_chunk_df["question"].notnull()
]
```
%% Cell type:markdown id: tags:
The LLM has generated three questions per chunk. Let's take a quick look.
%% Cell type:code id: tags:
``` python
questions_with_document_chunk_df.head(10)
```
%% Cell type:markdown id: tags:
### Retrieval Evaluation
We are now prepared to perform our retrieval evaluations. We will execute the queries we generated in the previous step and verify whether or not that the correct context is retrieved.
%% Cell type:code id: tags:
``` python
# First things first, let's reset phoenix
px.close_app()
px.launch_app()
```
%% Cell type:code id: tags:
``` python
# loop over the questions and generate the answers
for _, row in questions_with_document_chunk_df.iterrows():
question = row["question"]
response_vector = query_engine.query(question)
print(f"Question: {question}\nAnswer: {response_vector.response}\n")
```
%% Cell type:markdown id: tags:
Now that we have executed the queries, we can start validating whether or not the RAG system was able to retrieve the correct context. Let's extract all the retrieved documents from the traces logged to phoenix. (For an in-depth explanation of how to export trace data from the phoenix runtime, consult the [docs](https://docs.arize.com/phoenix/how-to/extract-data-from-spans)).
%% Cell type:code id: tags:
``` python
from phoenix.session.evaluation import get_retrieved_documents
retrieved_documents_df = get_retrieved_documents(px.Client())
retrieved_documents_df
```
%% Cell type:markdown id: tags:
Let's now use Phoenix's LLM Evals to evaluate the relevance of the retrieved documents with regards to the query. Note, we've turned on `explanations` which prompts the LLM to explain it's reasoning. This can be useful for debugging and for figuring out potential corrective actions.
%% Cell type:code id: tags:
``` python
from phoenix.evals import (
RelevanceEvaluator,
run_evals,
)
relevance_evaluator = RelevanceEvaluator(OpenAIModel(model="gpt-4-turbo-preview"))
retrieved_documents_relevance_df = run_evals(
evaluators=[relevance_evaluator],
dataframe=retrieved_documents_df,
provide_explanation=True,
concurrency=20,
)[0]
```
%% Cell type:code id: tags:
``` python
retrieved_documents_relevance_df.head()
```
%% Cell type:markdown id: tags:
We can now combine the documents with the relevance evaluations to compute retrieval metrics. These metrics will help us understand how well the RAG system is performing.
%% Cell type:code id: tags:
``` python
documents_with_relevance_df = pd.concat(
[retrieved_documents_df, retrieved_documents_relevance_df.add_prefix("eval_")], axis=1
)
documents_with_relevance_df
```
%% Cell type:markdown id: tags:
Let's compute Normalized Discounted Cumulative Gain [NCDG](https://en.wikipedia.org/wiki/Discounted_cumulative_gain) at 2 for all our retrieval steps. In information retrieval, this metric is often used to measure effectiveness of search engine algorithms and related applications.
%% Cell type:code id: tags:
``` python
import numpy as np
from sklearn.metrics import ndcg_score
def _compute_ndcg(df: pd.DataFrame, k: int):
"""Compute NDCG@k in the presence of missing values"""
n = max(2, len(df))
eval_scores = np.zeros(n)
doc_scores = np.zeros(n)
eval_scores[: len(df)] = df.eval_score
doc_scores[: len(df)] = df.document_score
try:
return ndcg_score([eval_scores], [doc_scores], k=k)
except ValueError:
return np.nan
ndcg_at_2 = pd.DataFrame(
{"score": documents_with_relevance_df.groupby("context.span_id").apply(_compute_ndcg, k=2)}
)
```
%% Cell type:code id: tags:
``` python
ndcg_at_2
```
%% Cell type:markdown id: tags:
Let's also compute precision at 2 for all our retrieval steps.
%% Cell type:code id: tags:
``` python
precision_at_2 = pd.DataFrame(
{
"score": documents_with_relevance_df.groupby("context.span_id").apply(
lambda x: x.eval_score[:2].sum(skipna=False) / 2
)
}
)
```
%% Cell type:code id: tags:
``` python
precision_at_2
```
%% Cell type:markdown id: tags:
Lastly, let's compute whether or not a correct document was retrieved at all for each query (e.g. a hit)
%% Cell type:code id: tags:
``` python
hit = pd.DataFrame(
{
"hit": documents_with_relevance_df.groupby("context.span_id").apply(
lambda x: x.eval_score[:2].sum(skipna=False) > 0
)
}
)
```
%% Cell type:markdown id: tags:
Let's now view the results in a combined dataframe.
%% Cell type:code id: tags:
``` python
retrievals_df = px.Client().get_spans_dataframe("span_kind == 'RETRIEVER'")
rag_evaluation_dataframe = pd.concat(
[
retrievals_df["attributes.input.value"],
ndcg_at_2.add_prefix("ncdg@2_"),
precision_at_2.add_prefix("precision@2_"),
hit,
],
axis=1,
)
rag_evaluation_dataframe
```
%% Cell type:markdown id: tags:
### Observations
Let's now take our results and aggregate them to get a sense of how well our RAG system is performing.
%% Cell type:code id: tags:
``` python
# Aggregate the scores across the retrievals
results = rag_evaluation_dataframe.mean(numeric_only=True)
results
```
%% Cell type:markdown id: tags:
As we can see from the above numbers, our RAG system is not perfect, there are times when it fails to retrieve the correct context within the first two documents. At other times the correct context is included in the top 2 results but non-relevant information is also included in the context. This is an indication that we need to improve our retrieval strategy. One possible solution could be to increase the number of documents retrieved and then use a more sophisticated ranking strategy (such as a reranker) to select the correct context.
%% Cell type:markdown id: tags:
We have now evaluated our RAG system's retrieval performance. Let's send these evaluations to Phoenix for visualization. By sending the evaluations to Phoenix, you will be able to view the evaluations alongside the traces that were captured earlier.
%% Cell type:code id: tags:
``` python
from phoenix.trace import DocumentEvaluations, SpanEvaluations
px.Client().log_evaluations(
SpanEvaluations(dataframe=ndcg_at_2, eval_name="ndcg@2"),
SpanEvaluations(dataframe=precision_at_2, eval_name="precision@2"),
DocumentEvaluations(dataframe=retrieved_documents_relevance_df, eval_name="relevance"),
)
```
%% Cell type:markdown id: tags:
### Response Evaluation
The retrieval evaluations demonstrates that our RAG system is not perfect. However, it's possible that the LLM is able to generate the correct response even when the context is incorrect. Let's evaluate the responses generated by the LLM.
%% Cell type:code id: tags:
``` python
from phoenix.session.evaluation import get_qa_with_reference
qa_with_reference_df = get_qa_with_reference(px.Client())
qa_with_reference_df
```
%% Cell type:markdown id: tags:
Now that we have a dataset of the question, context, and response (input, reference, and output), we now can measure how well the LLM is responding to the queries. For details on the QA correctness evaluation, see the [LLM Evals documentation](https://docs.arize.com/phoenix/llm-evals/running-pre-tested-evals/q-and-a-on-retrieved-data).
%% Cell type:code id: tags:
``` python
from phoenix.evals import (
HallucinationEvaluator,
OpenAIModel,
QAEvaluator,
run_evals,
)
qa_evaluator = QAEvaluator(OpenAIModel(model="gpt-4-turbo-preview"))
hallucination_evaluator = HallucinationEvaluator(OpenAIModel(model="gpt-4-turbo-preview"))
qa_correctness_eval_df, hallucination_eval_df = run_evals(
evaluators=[qa_evaluator, hallucination_evaluator],
dataframe=qa_with_reference_df,
provide_explanation=True,
concurrency=20,
)
```
%% Cell type:code id: tags:
``` python
qa_correctness_eval_df.head()
```
%% Cell type:code id: tags:
``` python
hallucination_eval_df.head()
```
%% Cell type:markdown id: tags:
#### Observations
Let's now take our results and aggregate them to get a sense of how well the LLM is answering the questions given the context.
%% Cell type:code id: tags:
``` python
qa_correctness_eval_df.mean(numeric_only=True)
```
%% Cell type:code id: tags:
``` python
hallucination_eval_df.mean(numeric_only=True)
```
%% Cell type:markdown id: tags:
Our QA Correctness score of `0.91` and a Hallucinations score `0.05` signifies that the generated answers are correct ~91% of the time and that the responses contain hallucinations 5% of the time - there is room for improvement. This could be due to the retrieval strategy or the LLM itself. We will need to investigate further to determine the root cause.
%% Cell type:markdown id: tags:
Since we have evaluated our RAG system's QA performance and Hallucinations performance, let's send these evaluations to Phoenix for visualization.
%% Cell type:code id: tags:
``` python
from phoenix.trace import SpanEvaluations
px.Client().log_evaluations(
SpanEvaluations(dataframe=qa_correctness_eval_df, eval_name="Q&A Correctness"),
SpanEvaluations(dataframe=hallucination_eval_df, eval_name="Hallucination"),
)
```
%% Cell type:markdown id: tags:
We now have sent all our evaluations to Phoenix. Let's go to the Phoenix application and view the results! Since we've sent all the evals to Phoenix, we can analyze the results together to make a determination on whether or not poor retrieval or irrelevant context has an effect on the LLM's ability to generate the correct response.
%% Cell type:code id: tags:
``` python
print("phoenix URL", px.active_session().url)
```
%% Cell type:code id: tags:
``` python
px.close_app()
```
%% Cell type:markdown id: tags:
## Conclusion
We have explored how to build and evaluate a RAG pipeline using LlamaIndex and Phoenix, with a specific focus on evaluating the retrieval system and generated responses within the pipelines.
Phoenix offers a variety of other evaluations that can be used to assess the performance of your LLM Application. For more details, see the [LLM Evals](https://docs.arize.com/phoenix/llm-evals/llm-evals) documentation.
evaluateRAG.py 0 → 100644
+ 354
− 0
View file @ 194bc3a2
import os
import subprocess
import json
from pathlib import Path
# enables counting list items
from operator import length_hint
# .env parser
from dotenv import load_dotenv
# getpass enables secure password input
from getpass import getpass
# creating temporary files
import tempfile
# download files containing the domain specific information
from urllib.request import urlparse, urlretrieve
# pandas handles table data
import pandas as pd
# The nest_asyncio module enables the nesting of asynchronous functions within an already running async loop.
import nest_asyncio
nest_asyncio.apply()
# colored print
from colorist import Color, BrightColor, bright_yellow, magenta, red, green
# phoenix is the framework & webservice from arize (https://docs.arize.com/phoenix)
import phoenix as px
from phoenix.evals import OpenAIModel, llm_generate, HallucinationEvaluator, QAEvaluator, run_evals
from phoenix.session.evaluation import get_retrieved_documents, get_qa_with_reference
from phoenix.evals import RelevanceEvaluator, run_evals
from phoenix.trace import DocumentEvaluations, SpanEvaluations
# llama index boilerplates chunking, verctorizing, storing querying aso. of private data
from llama_index.core import set_global_handler, SimpleDirectoryReader, VectorStoreIndex
from llama_index.core.node_parser import SimpleNodeParser
from llama_index.llms.openai import OpenAI
import numpy as np
from sklearn.metrics import ndcg_score
##################################################
### PREPARING & STARTING
##################################################
##########
# SET VARS
##########
# Load environment variables from .env file
if not os.path.isfile('./.env'):
raise RuntimeError("Aborting: No .env file found.")
load_dotenv()
# tell llama_index to send all infos to the phoenix instance
set_global_handler("arize_phoenix")
# ##########
# # LOAD OPENAPI KEY
# ##########
if not (openai_api_key := os.getenv("OPENAI_API_KEY")):
openai_api_key = getpass("🔑 Enter your OpenAI API key: ")
os.environ["OPENAI_API_KEY"] = openai_api_key
# Define the LLM
llm = OpenAI(model="gpt-4")
##########
# START PHOENIX
##########
# check for running process
process = subprocess.run(["lsof", f"-iTCP:{os.environ['PHOENIX_PORT']}"], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
output = process.stdout.decode("utf8")
if len(output.strip()) > 1:
print(f"Aborting: Error while attempting to bind on address ('0.0.0.0', {os.environ['PHOENIX_PORT']}): address already in use")
os._exit(1)
# launch phoenix
green(f"Launching phoenix application")
session = px.launch_app()
("phoenix URL", px.active_session().url)
##################################################
### PREPARING FILES FOR RAG
##################################################
##########
# LOAD DOMAIN SPECIFIC INFORMATION FILES
##########
# create array of files to download
urls = os.environ['DOMAIN_DOCS'].split(',')
# create download folder
Path(os.environ['DOMAIN_DOWNLOAD_FOLDER']).mkdir(parents=True, exist_ok=True)
# for each file to download
for url in urls:
# parse url
source = urlparse(os.environ['DOMAIN_BASEPATH'] + url)
# transfer path into filename (because filenames may occur multiple times in different paths)
targetFilename = source.path.replace('/',':')
# remember full target path
targetPath = os.environ['DOMAIN_DOWNLOAD_FOLDER']+'/'+ targetFilename
# output
print(f'downloading {source.path} \t>\t {targetPath}')
# download
urlretrieve(os.environ['DOMAIN_BASEPATH'] + url, targetPath)
# load docs into llamaIndex
documents = SimpleDirectoryReader(input_dir=os.environ['DOMAIN_DOWNLOAD_FOLDER']).load_data()
##########
# CHUNK & VECTORIZE FILES
##########
green(f"Building index with a chunk_size of {os.environ['DOCUMENTS_CHUNK_SIZE']} for {length_hint(documents)} docs")
# define parsing options incl. regex, paragraph_separator aso.
node_parser = SimpleNodeParser.from_defaults(chunk_size=int(os.environ['DOCUMENTS_CHUNK_SIZE']))
# generate nodes based on the parser
nodes = node_parser.get_nodes_from_documents(documents)
# Vectorize nodes
# very first time OPENAI_API_KEY is needed in this script
vector_index = VectorStoreIndex(nodes)
print(f'created {length_hint(nodes)} chunks')
countQuestions = length_hint(nodes) * int(os.environ['QUESTIONS_PER_CHUNK'])
# # Build a QueryEngine and start querying.
query_engine = vector_index.as_query_engine()
##########
# CREATE THE QUESTION-CONTEXT PAIRS
##########
green(f'Generating {countQuestions} questions. {os.environ['QUESTIONS_PER_CHUNK']} per chunk')
# create a dataframe of all the document chunks that were indexed
document_chunks_df = pd.DataFrame({"text": [node.get_text() for node in nodes]})
# template to generate {QUESTIONS_PER_CHUNK} questions per chunk
generate_questions_template = f"""\
Context information is below.
---------------------
{{text}}
---------------------
Given the context information and not prior knowledge.
generate only questions based on the below query.
You are a Teacher/ Professor. Your task is to setup \
{os.environ['QUESTIONS_PER_CHUNK']} questions for an upcoming \
quiz/examination. The questions should be diverse in nature \
across the document. Restrict the questions to the \
context information provided."
Output the questions in JSON format with the keys question_1, question_2, question_3.
"""
# define parser to fetch questions from the response
def output_parser(response: str, index: int):
try:
return json.loads(response)
except json.JSONDecodeError as e:
return {"__error__": str(e)}
# prompt template to LLM and store > questions_df
questions_df = llm_generate(
dataframe=document_chunks_df,
template=generate_questions_template,
model=OpenAIModel(
model="gpt-3.5-turbo",
),
output_parser=output_parser,
concurrency=20,
)
# Construct a dataframe of the questions and the document chunks
questions_with_document_chunk_df = pd.concat([questions_df, document_chunks_df], axis=1)
questions_with_document_chunk_df = questions_with_document_chunk_df.melt(
id_vars=["text"], value_name="question"
).drop("variable", axis=1)
# If the above step was interrupted, there might be questions missing. Clean up the dataframe.
questions_with_document_chunk_df = questions_with_document_chunk_df[
questions_with_document_chunk_df["question"].notnull()
]
# debug print
# magenta(questions_with_document_chunk_df.head(10))
##################################################
### RETRIEVAL EVALUATION
##################################################
green('Starting retrieval evaluation')
##########
# GENERATE THE ANSWERS
########
print(f'Generating the answers for each question')
# loop over the questions and generate the answers
for _, row in questions_with_document_chunk_df.iterrows():
question = row["question"]
response_vector = query_engine.query(question)
# debug print question-answer pair
# print(f"Question: {Color.MAGENTA}{question}{Color.OFF}\nAnswer: {BrightColor.MAGENTA}{response_vector.response}{Color.OFF}\n")
# extract all the retrieved documents from the traces logged to phoenix
print('extracting the retrieved documents from phoenix traces')
retrieved_documents_df = get_retrieved_documents(px.Client())
##########
# CALCULATE RELEVANCE
########
# use Phoenix's LLM Evals to evaluate the relevance of the retrieved documents with regards to the query.
# Note the turned on explanations which prompts the LLM to explain it's reasoning.
# This can be useful for debugging and for figuring out potential corrective actions.
print('calculating relevance of the documents to the query')
relevance_evaluator = RelevanceEvaluator(OpenAIModel(model="gpt-4-turbo-preview"))
retrieved_documents_relevance_df = run_evals(
evaluators=[relevance_evaluator],
dataframe=retrieved_documents_df,
provide_explanation=True,
concurrency=20,
)[0]
# combine the documents with the relevance evaluations to compute retrieval metrics.
# These metrics will help to understand how well the RAG system is performing.
documents_with_relevance_df = pd.concat(
[retrieved_documents_df, retrieved_documents_relevance_df.add_prefix("eval_")], axis=1
)
##########
# NCDG@2
##########
print(f'Computing NCDG@2')
# function
def _compute_ndcg(df: pd.DataFrame, k: int):
"""Compute NDCG@k in the presence of missing values"""
n = max(2, len(df))
eval_scores = np.zeros(n)
doc_scores = np.zeros(n)
eval_scores[: len(df)] = df.eval_score
doc_scores[: len(df)] = df.document_score
try:
return ndcg_score([eval_scores], [doc_scores], k=k)
except ValueError:
return np.nan
# run
ndcg_at_2 = pd.DataFrame(
{"score": documents_with_relevance_df.groupby("context.span_id").apply(_compute_ndcg, k=2)}
)
##########
# PRECISION@2
##########
print(f'Computing Precision@2')
precision_at_2 = pd.DataFrame(
{
"score": documents_with_relevance_df.groupby("context.span_id").apply(
lambda x: x.eval_score[:2].sum(skipna=False) / 2
)
}
)
##########
# HIT
##########
print(f'Computing HIT')
hit = pd.DataFrame(
{
"hit": documents_with_relevance_df.groupby("context.span_id").apply(
lambda x: x.eval_score[:2].sum(skipna=False) > 0
)
}
)
##########
# COMBINE METRICS INTO ONE DATAFRAME
##########
print(f'Creating dataframe of all metrics')
retrievals_df = px.Client().get_spans_dataframe("span_kind == 'RETRIEVER'")
rag_evaluation_dataframe = pd.concat(
[
retrievals_df["attributes.input.value"],
ndcg_at_2.add_prefix("ncdg@2_"),
precision_at_2.add_prefix("precision@2_"),
hit,
],
axis=1,
)
# debug
# magenta(rag_evaluation_dataframe)
##########
# AGGREGATE THE SCORES ACROSS THE RETRIEVALS
##########
print(f'Aggregated metrics are:')
results = rag_evaluation_dataframe.mean(numeric_only=True)
magenta(results)
# SEND PERFORMANCE INFO TO PHOENIX
px.Client().log_evaluations(
SpanEvaluations(dataframe=ndcg_at_2, eval_name="ndcg@2"),
SpanEvaluations(dataframe=precision_at_2, eval_name="precision@2"),
DocumentEvaluations(dataframe=retrieved_documents_relevance_df, eval_name="relevance"),
)
##################################################
### RESPONSE EVALUATION
##################################################
green(f'Starting response evaluation')
# fetching question, context, and response (input, reference, and output) into one dataframe
qa_with_reference_df = get_qa_with_reference(px.Client())
# debug print
# magenta(qa_with_reference_df)
##########
# CALCULATE CORRECTNESS & HALLUCINATIONS
##########
# measure how well the LLM is responding to the queries
# details: https://docs.arize.com/phoenix/evaluation/how-to-evals/running-pre-tested-evals/q-and-a-on-retrieved-data
qa_evaluator = QAEvaluator(OpenAIModel(model="gpt-4-turbo-preview"))
hallucination_evaluator = HallucinationEvaluator(OpenAIModel(model="gpt-4-turbo-preview"))
print('calculate performance of QA correctness & hallucinations')
qa_correctness_eval_df, hallucination_eval_df = run_evals(
evaluators=[qa_evaluator, hallucination_evaluator],
dataframe=qa_with_reference_df,
provide_explanation=True,
concurrency=20,
)
# debug
# magenta(qa_correctness_eval_df.head())
# magenta(hallucination_eval_df.head())
# aggregate results to get a sense of how well the LLM is answering the questions given the context
qa_correctness_eval_df.mean(numeric_only=True)
hallucination_eval_df.mean(numeric_only=True)
# send QA performance and Hallucinations performance to Phoenix for visualization
print('send aggregated evaluations to phoenix')
px.Client().log_evaluations(
SpanEvaluations(dataframe=qa_correctness_eval_df, eval_name="Q&A Correctness"),
SpanEvaluations(dataframe=hallucination_eval_df, eval_name="Hallucination"),
)
##################################################
### CLEAN END
##################################################
green('finished evaluation process')
print("See result here: ", px.active_session().url)
bright_yellow("Press Enter to exit...")
input()
px.close_app()
##################################################
### SHITTY SNIPPETS, NEVER SUPPOSED TO RUN
##################################################
# if process.returncode != 0:
# print(process.stderr.decode("utf8"))
# raise RuntimeError(f"Returncode {process.returncode} Und das war nicht gut und der error ist ")
# print(process.returncode)
# print(process.stdout.decode("utf8"))
# output = process.stdout.decode("utf8")
# if len(output.strip()) > 1:
# print("ham wir")
# else:
# print("ham wir nicht")
# print(process.stderr.decode("utf8"))
requirements.txt 0 → 100644
+ 128
− 0
View file @ 194bc3a2
aiohttp==3.9.4
aiosignal==1.3.1
annotated-types==0.6.0
anyio==4.3.0
arize-phoenix==3.21.0
arize-phoenix-evals==0.7.0
attrs==23.2.0
beautifulsoup4==4.12.3
cachetools==5.3.3
certifi==2024.2.2
charset-normalizer==3.3.2
click==8.1.7
colorist==1.7.2
Cython==0.29.37
dataclasses-json==0.6.4
decorator==5.1.1
Deprecated==1.2.14
dirtyjson==1.0.8
distro==1.9.0
frozenlist==1.4.1
fsspec==2024.3.1
gcsfs==2024.3.1
google-api-core==2.18.0
google-auth==2.29.0
google-auth-oauthlib==1.2.0
google-cloud-core==2.4.1
google-cloud-storage==2.16.0
google-crc32c==1.5.0
google-resumable-media==2.7.0
googleapis-common-protos==1.63.0
graphql-core==3.2.3
greenlet==3.0.3
grpcio==1.62.1
h11==0.14.0
hdbscan==0.8.33
httpcore==1.0.5
httpx==0.27.0
idna==3.7
importlib-metadata==7.0.0
Jinja2==3.1.3
joblib==1.4.0
llama-index==0.10.28
llama-index-agent-openai==0.2.2
llama-index-callbacks-arize-phoenix==0.1.3
llama-index-cli==0.1.11
llama-index-core==0.10.28
llama-index-embeddings-openai==0.1.7
llama-index-indices-managed-llama-cloud==0.1.5
llama-index-legacy==0.9.48
llama-index-llms-openai==0.1.15
llama-index-multi-modal-llms-openai==0.1.5
llama-index-program-openai==0.1.5
llama-index-question-gen-openai==0.1.3
llama-index-readers-file==0.1.17
llama-index-readers-llama-parse==0.1.4
llama-parse==0.4.0
llamaindex-py-client==0.1.18
llvmlite==0.42.0
MarkupSafe==2.1.5
marshmallow==3.21.1
multidict==6.0.5
mypy-extensions==1.0.0
nest-asyncio==1.6.0
networkx==3.3
nltk==3.8.1
numba==0.59.1
numpy==1.26.4
oauthlib==3.2.2
openai==1.17.1
openinference-instrumentation==0.1.1
openinference-instrumentation-langchain==0.1.14
openinference-instrumentation-llama-index==1.2.1
openinference-instrumentation-openai==0.1.4
openinference-semantic-conventions==0.1.5
opentelemetry-api==1.24.0
opentelemetry-exporter-otlp==1.24.0
opentelemetry-exporter-otlp-proto-common==1.24.0
opentelemetry-exporter-otlp-proto-grpc==1.24.0
opentelemetry-exporter-otlp-proto-http==1.24.0
opentelemetry-instrumentation==0.45b0
opentelemetry-proto==1.24.0
opentelemetry-sdk==1.24.0
opentelemetry-semantic-conventions==0.45b0
packaging==24.0
pandas==2.2.2
pillow==10.3.0
proto-plus==1.23.0
protobuf==4.25.3
psutil==5.9.8
pyarrow==15.0.2
pyasn1==0.6.0
pyasn1_modules==0.4.0
pydantic==2.7.0
pydantic_core==2.18.1
pynndescent==0.5.12
pypdf==4.2.0
python-dateutil==2.9.0.post0
python-dotenv==1.0.1
pytz==2024.1
PyYAML==6.0.1
regex==2023.12.25
requests==2.31.0
requests-oauthlib==2.0.0
rsa==4.9
scikit-learn==1.4.2
scipy==1.13.0
setuptools==69.2.0
six==1.16.0
sniffio==1.3.1
sortedcontainers==2.4.0
soupsieve==2.5
SQLAlchemy==2.0.29
starlette==0.37.2
strawberry-graphql==0.208.2
striprtf==0.0.26
tenacity==8.2.3
threadpoolctl==3.4.0
tiktoken==0.6.0
tqdm==4.66.2
typing-inspect==0.9.0
typing_extensions==4.11.0
tzdata==2024.1
umap-learn==0.5.6
urllib3==2.2.1
uvicorn==0.29.0
wrapt==1.16.0
yarl==1.9.4
zipp==3.18.1
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Please register or to comment