We are building an LLM-based application that takes a lot of user data from various internal sources and sends the data to various prompts which provide the answers needed to fill out forms correctly.
The system used to be more basic, where a single LLM prompt would answer a question. The prompts were stored in a database so that metrics related to the prompt could be recorded and used for future improvements. The outputs of the prompt were uniform, allowing prompt engineers to modify the prompts and release new improved prompts usually without any code changes.
As we are moving on and the ability of LLMs has changed, we are finding that a complex chain of prompts with various associated schemas for each prompt will give better results to the end questions that are being asked. The prompt chains may be simple feeds from on to the other or conditional where based on the output different prompts may be called.
This approach requires a more complex prompt management system and the question is should prompts be stored in git or the DB?
If we store in git, it allows changes to the prompt chains and logic to be handled easily and new types of prompt chains/flows to be easily implemented as the need arises. However, we lose the ability to allow quick updates of prompts without code updates and tracking metrics and comparing evaluations of two different prompts is more complicated when it has to be related back to a specific code version/git commit.
On the flip side, storing multiple prompts that go together in a chain in the DB and creating a codebase that is robust enough to allow for updates of all these different types of chains can be complex.
What would be the recommended approach and the most efficient architecture?