Ingestion Provider

Ingestion providers determine how your documents are processed when they are uploaded. The available options are:

• Globant Enterprise AI (Default)
• llamaParse
• legacy

To set up an ingestion provider, go to the RAG Assistants home page and click on ADD DOCUMENTS. Next, select the Advanced tab and configure the ingestion provider according to your needs.

Globant Enterprise AI (geai)

The geia option allows you to take advantage of the information contained in the images and tables of PDFs, not only processing the raw text but also the images and tables (depending on the selected strategy). This ensures you make the most of all the content of the documents (scanned and standard PDF files are currently supported).

The geai option processes files differently based on their content. For PDF files they can be:

• Scanned PDF where each page is treated as an image and will use by default the hi_res strategy and the scannedPrompt parameter. Therefore, for each page an LLM call will be executed to interpret the image and get as much text as possible.
• Standard PDF containing text, images and tables will take into account the selected strategy parameter with a combination of imagePrompt/tablePrompt parameters.

Parameters

You can customize the geai processing with these parameters:

Default imagePrompt

The imagePrompt default is as follows:

You are an assistant tasked with extracting all text from images in the image text language.
These images are pages from a PDF document. Extract and transcribe all visible text in the image,
maintaining the structure and layout as much as possible. Include any headers, footers, and page numbers.
Be thorough and don't miss any text, no matter how small or where it's positioned in the image.

Default tablePrompt

The tablePrompt default is as follows:

You are an assistant tasked with extracting all text from images in the image text language. 
These images are pages from a PDF document. Extract and transcribe all visible text in the image, 
maintaining the structure and layout as much as possible. Include any headers, footers, and page numbers.
Be thorough and don't miss any text, no matter how small or where it's positioned in the image.

Default scannedPrompt

The scannedPrompt default is as follows:

You are an assistant tasked with extracting all text from images in the image text language. 
These images are pages from a PDF document. Extract and transcribe all visible text in the image, 
maintaining the structure and layout as much as possible. Include any headers, footers, and page numbers.
Be thorough and don't miss any text, no matter how small or where it's positioned in the image.

Samples

The following shows how to use Chat API for different ingestion provider options:

• Minimal options:

curl -X POST "$BASE_URL/v1/search/profile/{name}/document" \
 -H "Authorization: Bearer $SAIA_PROJECT_APITOKEN" \
 -H "Content-Type: multipart/form-data" \
 --form 'file=@"/C:/temp/SampleFile.pdf"' \
 --form 'provider="geai"'

• High-resolution strategy:

 
curl -X POST "$BASE_URL/v1/search/profile/{name}/document" \
 -H "Authorization: Bearer $SAIA_PROJECT_APITOKEN" \
 -H "Content-Type: multipart/form-data" \
 --form 'file=@"/C:/temp/SampleFile.pdf"' \
 --form 'provider="geai"' \
 --form 'strategy="hi_res"'
 

• High-resolution with custom model and prompt:

curl -X POST "$BASE_URL/v1/search/profile/{name}/document" \
 -H "Authorization: Bearer $SAIA_PROJECT_APITOKEN" \
 -H "Content-Type: multipart/form-data" \
 --form 'file=@"/C:/temp/SampleFile.pdf"' \
 --form 'provider="geai"' \
 --form 'model="openai/gpt-4o"' \
 --form 'imagePrompt="Resume the image as succinctly as possible in markdown format"'

llamaParse

LlamaParse is an API by LlamaIndex to parse and represent files for efficient retrieval and context augmentation.

When selecting the llamaParse option, ingestion will redirect to get text representation of the associated documents; you need an API Key to use the service (check pricing and usage).

The following parameters are available:

Considerations

• pageNumber parameter is not assigned when ingesting PDF files.

Samples

The following shows how to use Chat API for different ingestion provider options:

• Markdown format, first page only:

curl -X POST "$BASE_URL/v1/search/profile/{name}/document" \
 -H "Authorization: Bearer $SAIA_PROJECT_APITOKEN" \
 -H "Content-Type: multipart/form-data" \
 --form 'file=@"/C:/temp/SampleFile.pdf"' \
 --form 'provider="llamaParse"' \
 --form 'format="markdown"' \
 --form 'targetPages="0"' \
 --form 'splitByPage="true"' \
 --form 'apiKey=""'

• Text format without splitting, two languages:

curl -X POST "$BASE_URL/v1/search/profile/{name}/document" \
 -H "Authorization: Bearer $SAIA_PROJECT_APITOKEN" \
 -H "Content-Type: multipart/form-data" \
 --form 'file=@"/C:/temp/SampleFile.pdf"' \
 --form 'provider="llamaParse"' \
 --form 'format="text"' \
 --form 'splitByPage="false"' \
 --form 'language="en"' \
 --form 'language="es"'

Note: The "name" in the URL refers to the associated RAG Assistant identifier.

legacy

The legacy option uses the following packages:

• officeparser version 4.1.1 or higher
• pdf-parse version 1.1.1 or higher.

Note: This is the fastest option, but its accuracy may vary depending on the file types.

Custom Ingestion (ETL/ELT)

If you need a custom ingestion process, you can create your own ETL (Extract/Transform/Load) or ELT (Extract/Load/Transform) extraction process and then upload a .custom file. This allows you to take complete control of the process by creating the necessary Document Interface for correct processing.

This option is useful when you need more control over how your data is extracted, transformed, and loaded into the system.

Ingestion samples

• legacy vs geai
• table structure ingestion strategy
• Documents with images hi_res strategy
• Documents with tables hi_res strategy
• scanned documents processing

See Also

Ingestion SDK