File Support in Assistants allows you to work with various file types, including documents, images, videos, and audio, to perform tasks like summarizing documents, extracting information, comparing files, and generating text based on file content. This opens up a world of possibilities for using assistants in new and innovative ways.

Scenarios

The end user should be able to interact with assistants using files in the following ways:

• Summarize a document: Provide a concise summary of the document's content.
• Extract specific information: Extract specific data points from a document, such as the amount, client name, and deadline from a contract.
• Compare documents: Compare the information in two documents, identifying matching and differing clauses.
• Generate text based on document content: Create new text based on the content of a document.
• Ask questions based on uploaded documents: Similar to RAG Assistants, but with the ability to upload new documents from the Playground Interface to add to the assistant's context.

Functionality

File Support in Assistants provides a robust set of features for managing and using files:

• File Management: You can manage files associated with each project using CRUD operations (create, read, update, delete). A project-level API is available for file management, using the project identifier as the key. You can also configure a storage limit per project to manage storage space effectively.
• File Referencing: You can reference files within assistant prompts using variables like {file:filename} or {file:*}. For example, you can use the prompt: "Compare {file:id1} with {file:id2}". The system seamlessly handles file uploads, context window size, and summarization, regardless of whether the model is multimodal or not.
• File Types: File support extends beyond documents to include images and videos, using a similar syntax: {file:image1} or {file:video1}. For example, you can use the prompt: "Extract the text from image {file:id1} and summarize its content".
• Frontend Upload: End users can easily select existing files or upload new ones when interacting with file-handling assistants.

How to Use File Support in Assistants

1. Choose your assistant: Select the assistant that best suits your needs.
2. Upload your file: Click on the “Upload File” button and select the file you want to use. The assistant will recognize the file type and process it accordingly.
3. Write your prompt: Enter your instructions for the assistant in the prompt box. Be clear and specific about what you want the assistant to do.
4. Use file references: To reference the uploaded file in your prompt, use the following syntax:
   • Documents: {file:filename} (for example, {file:contract})
   • Images: {file:image1}
   • Videos: {file:video1}
   • Audio: {file:audio1}
5. Submit your prompt: Click on the arrow icon (or press Enter on your keyboard) to send your prompt to the assistant.
6. Review the results: The assistant will process your request and display the results. 

Samples

Document:

• Prompt: “Summarize the key points of the document {file:contract}”.
• Response: “The contract outlines the terms of service for a software subscription, including pricing, payment terms, and support options”.

Image:

• Prompt: “Describe the image {file:product_image}”.
• Response: “The image shows a red sports car with black leather interior”.

Video:

• Prompt: “Summarize the main points of the video {file:tutorial}”.
• Response: “The video demonstrates how to use the new software feature for creating custom reports”.

Audio:

• Prompt: “Transcribe the audio file {file:meeting_recording}”.
• Response: The assistant will provide a text transcript of the audio recording.