Promptfoo

What it is

Promptfoo\nPromptfoo is a library designed for Language Model Mathematics (LLM) prompt testing and evaluation. It allows users to create a list of test cases using a representative sample of user inputs, minimizing subjectivity in prompt fine-tuning.\n\nSecond paragraph

Features

• Test Case Creation—Users can create a list of test cases using a representative sample of user inputs, minimizing subjectivity in prompt fine-tuning.
• Evaluation Metrics—Promptfoo offers flexibility in setting up evaluation metrics. Users can choose from built-in metrics or define custom ones to meet specific needs.
• Prompt and Model Comparison—Users can conveniently compare prompts and model outputs side-by-side, facilitating prompt and model selection.
• Integration-friendly—The library seamlessly integrates into existing testing or continuous integration (CI) workflows, ensuring a smooth and efficient process.
• Web Viewer and CLI—Promptfoo offers both a web viewer and a command-line interface, catering to different user preferences and needs.
• Proven Trustworthiness—Trusted by LLM applications serving over 10 million users, Promptfoo demonstrates its reliability and popularity within the LLM community.

Pricing

Pricing model: Free