Deployment Questions for Streamlit Cloud & Introducing PolyRAG (RAG Fork)

[QuentinFuxa]

Hi @JoshuaC215

First off, a huge thank you for creating the template! It's incredibly well-structured and helpful

Inspired by your work, I've created a fork called PolyRAG (https://github.com/QuentinFuxa/PolyRAG). My fork extends the toolkit to specifically focus on Retrieval-Augmented Generation (RAG), enabling agents to interact with both PostgreSQL databases (with dynamic schema discovery) and PDF documents (using backends like nlm-ingestor or pymupdf for indexing and vector/text search). It also includes features like direct PDF uploads, Plotly visualizations, and conversation history management within PostgreSQL

I'm planning to deploy a public demo using Streamlit Community Cloud, similar to your demo app, to showcase PolyRAG's capabilities, with a demo set of documents and a small postgres db attached. However, I've run into a couple of questions regarding the deployment strategy, and seeing that you have a running Streamlit demo, I was hoping you might share some insights:

1. Backend hosting: How and where do you host the FastAPI backend service (run_service.py) for your public Streamlit demo? Since Streamlit Community Cloud only hosts the streamlit app itself, I assume the agent service needs to run elsewhere (e.g., Render, Heroku, AWS, GCP, etc.). Any recommendations or pointers on how you approached this would be greatly appreciated.
2. Api key management: For your public demo, how do you manage the open api key (or other necessary llm api keys)? Did you embed a key directly in the .env ? If so, what strategies do you use (or recommend) to prevent abuse and manage costs associated with public users making numerous queries?

Finally, as I continue to actively develop PolyRAG, would you be open to potentially adding a link to it in the "Projects built with or inspired by agent-service-toolkit" section of your README? I think it represents a specific, RAG-focused application of your toolkit that might be interesting to others in the community.

Thanks again for the great project and any guidance you can offer!

[JoshuaC215]

This is awesome!! Very nice!! Excited to try the demo.

To your questions:

1. Your assumption is correct. The agent-service-toolkit backend is hosted as an Azure Web App. You can see the deployment config in deploy.yml. Basically for each merge, I push to a public docker image repository and then update the image tag on the app. I set up the app manually in the Azure web portal. Honestly it took some time and experimentation to get working since it was my first time deploying in Azure, but I haven't touched it since I got it working and the hosting cost has been free. I expect that Heroku or AWS would work just fine as well. One caveat, I use a local sqlite for storage on the app container to keep the cost and complexity down, but it also means the db gets wiped periodically.

2. The Streamlit app secrets only contains the URL of the Azure backend and the AUTH_SECRET key to reduce the risk of abuse. The other ENV including the OpenAI API Key are attached to the backend. On the cost side I keep a fairly low billing limit on my OpenAI API Key and default users to gpt-4o-mini which is extremely cheap. In practice it ends up costing me $1-2/month.

Hope this helps! I will add your project to the README.

[QuentinFuxa]

Thanks a lot for the reply and for adding PolyRAG to the README!
Your tips on hosting and key management are super helpful. I’ll work on deploying PolyRAG thanks to your advice. Really appreciate that!
Quentin