Deep Dive into OpenAI Assistants Playground with the Retrieval tool
The Assistants API is poorly documented. This week I decided to do a forensic deep dive into the API using the Playground interface..
In November 2023, I started this new section, AI STUFF. The AI STUFF posts will alternate with my regular posts. The AI STUFF posts will be in English and Turkish. They are written in English and translated to Turkish. Both will be on the same page.
Kasım 2023 den itibaren yeni bir bölüme başlıyorum: AI STUFF. AI STUFF gönderileri normal gönderilerimle dönüşümlü olarak yayınlanacak ve ayni sayfada İngilizce ve Türkçe olacak.
From one-off questions moving to a chatbot
You will remember the Assistant named “Assistant for General” from last post. I create it in the file QA.IPYNB. In that post, I only used for single questions and single answers. It worked very well in that mode.
Later, I tried to provide a proper chat by keeping the discussion in the same thread. The OpenAI documentation says that this is all you need and the Assistants API will keep track of the previos Q&A and formulate the next answer accordingly.
Unfortunately, it did not work. Here is an example:
Example of API call failure
Query #1 : What is the local government debt in China?
The first stage in answering this question is locating the relevant files. This is a local job and done by my program. The top three most relevant pages are identified. They are:
https://halimgur.substack.com/p/retirement-a-journey-of-continued
https://halimgur.substack.com/p/will-there-be-a-war
https://halimgur.substack.com/p/rogue-renaissance-on-globe-with-upheavals
This is correct. The first file is actually the most relevant. The others are marginally related. I created a thread, attached these files, and asked the question. The answer was:
Answer #1: In 2020, China's local government debt approached 90 trillion yuan (approximately 12.49 trillion U.S. dollars), which equated to 88% of the country's GDP at that time.
This is the correct answer. I then asked the following question in the same thread:
Query #2: This is a good answer. Could you tell me who did this study?
OpenAI Assistant failed to answer this question. Its answer was:
Answer #2: I don’t know.
I tried again:
Question #3: I know the name of the Professor is in the same reference file. Could you check that file again and tell me what that name is?
This did not work either. The answer was a repetition of the first.
Answer #3: In 2020, China's local government debt approached 90 trillion yuan (approximately 12.49 trillion U.S. dollars), which equated to 88% of the country's GDP at that time【10†source】
Unfortunately, the Assistants API is not well-documented. There are no examples and even the API references are incomplete. I monitored the OpenAI forum. I did not see a similar problem posted. In fact, I did not see many postings on the Assistants API.
It is possible that people are waiting for OpenAI to make better documentation available, which may happen when the Store is announced in February.
Playground Success
I then repeated the same in the playground. I used the same Assistant. I uploaded the same three files to the Assistant and asked the same questions and even an extra question. It worked like a charm. Here are the questions and answers. Note that OpenAI stores the messages in reverse order. So the question at the top of the table is the last question.
These questions were provided with no extra prompting.
This posed a dillemma. The Assistant works very well when the API calls are placed through the Playground. The Assistant fails miserably when the same API calls are placed through my Jupyter notebook. The lack of documentation made it very difficult to progress this further without a better understanding of the Assistant API.
Thankfully, the Playground is transparent and provides the IDs of the runs and the threads. Using those IDs, it was possible to do a forensic analysis on how the Assistants API works. This analysis is presented in `playground.ipynb` in my GitHub probot repository.
The previous posts explain how you can download my files from GitHub. I hope you will find it useful.
In the next post when I will again try to build a chatbot, I will see if this has been a useful exercise for me.
Python ve OpenAI API kullanıyorum. Ek olarak, bir X (eski adıyla Twitter) Topluluk Grubu 'Yapı Yapay Zeka Eğitmenleri' oluşturdum. Eğer ilgileniyorsanız lütfen gruba katılın. Gelişmelerimi orada da paylaşacağım.
Deneyimli bir Python programcısı olmanıza gerek yok ancak biraz Python bilgisi faydalı olacaktır. Aşağıda bana katılmak için atmanız gereken adımları listeliyorum ve bunu nasıl yapacağınızı açıklıyorum:
VS Code'u https://code.visualstudio.com/download adresinden yükleyin. Bu ücretsiz bir program ve benim kullandığım araçtır, dolayısıyla sorun yaşamanız durumunda size yardımcı olabilirim.
Python'u https://www.python.org/downloads/ adresinden yükleyin.
Tüm program ve detaylı anlatımları github'daki hesabımdan indirin. Bunu yapmak için bir github hesabınızın olması gerektiğini sanmıyorum. Projenin adı probot. Probot klasörünün tamamını zip paketi olarak indirebilir ve zip dosyasını yerel diskinize açabilirsiniz