LangSmith alternatives: 5 tools to capture LLM data

LangSmith, developed by the LangChain team, is a powerful tool for observing and managing LangChain apps.

However, it has some limitations:

• Not open-source
• Limited compatibility with frameworks outside of LangChain
• Dashboard UI is not the best
• Expensive personalized support

Let's explore 5 viable alternatives to LangSmith that offer comparable functionalities and additional benefits.

Lunary

Lunary is a platform for AI devs that features almost all the cool stuff from LangSmith, plus:

• 100% open-source and self-hostable
• Designed for LLM Chatbots
• Fully compatible with LangChain and other libraries
• Clean and simple UI
• Easy to use prompt library
• Experiment with models such as Mistral, Claude v3, Gemini Pro
• New features every months
• Bootstrapped company that cares more about customers than investors

Lunary is free to use until you hit 1000 logs collected per day, which should be enough until you go into production.

PromptLayer

PromptLayer is another great tool more focused on managing prompts. It also offers prompt management as well as analytics.

However, PromptLayer pricing structure is not clear from looking at their website, and is not open-source which may be a deal-breaker for some projects.

Arize

Another big player in the ML observability game, Arize boasts a powerful set of tools around all steps of building a machine-learning application. Arize is not generally focused on LLMs but has recently taking a big turn in building LLM features.

The drawback with Arize is that it's more complicated to learn to use. It's also not open-source and offers only 1K logs per month in the free plan (which is 30x less than Lunary.)

Helicone

If your observability needs are simpler than what LangSmith offers, Helicone can be a great choice.

It's a simple, open-source tool that offers a clean UI and is easy to use.

Helicone originally works as a proxy, giving you a replacement URL for your OpenAI calls that automatically logs the requests and responses. It's a great way to get started with observability without having to add an extra library to your code.

Weights & Biases

Weights & Biases (known as Wandb) is a popular tool for MLOps that's been around for a while. It offers a wide range of features for LLMs, including tracing, logging, fine-tuning, evaluations, visualization, and collaboration tools.

The platform and integration might be a bit overwhelming to get accustomed to, but it's a great choice for ML-heavy projects that need absolutely everything in one place.

Note that it's not open-source, but it's free to use for small projects. The pricing is then done per hour which might be a bit confusing to relate to your usage.

For larger enterprises with big ML teams, it's a great choice. WandDB even used by the teams at OpenAI.

Conclusion

There are many great tools available for capturing and managing logs for your LLM applications.

If you're looking for a simple, open-source solution, Helicone is a great choice. For more advanced features: Lunary, and Weights & Biases are all excellent options.

Ultimately, the best tool for you will depend on your specific needs and budget. We recommend trying out a few different tools to see which one works best for you.