Live Chat With Tomรกs Hernando Kofman about NotDiamond!
Key Takeaways at a Glance
17:40
NotDiamond is an AI model router that optimizes query responses.20:41
The future of AI involves multiple specialized models.22:18
Open sourcing enhances the AI routing ecosystem.29:11
Developers can customize routing based on specific needs.29:28
NotDiamond offers low-latency model routing for applications.30:38
Integrating NotDiamond into existing applications is straightforward.32:41
NotDiamond supports a wide range of models and rapid onboarding.35:01
Model performance varies based on specific use cases.45:27
Leveraging LMS for customer support and HR tasks is effective.47:41
Incremental improvements in LMS can yield substantial benefits.48:40
Cost reduction while improving quality is achievable with LMS.56:03
Routing models can optimize performance across multiple languages.1:01:05
Custom routing can significantly enhance model performance.1:02:20
The name 'NotDiamond' reflects philosophical concepts.1:04:04
NotDiamond offers free queries to users.
1. NotDiamond is an AI model router that optimizes query responses.
๐ฅ92
17:40
NotDiamond automatically selects the best language model for a given query, enhancing accuracy while reducing costs and latency.
- It outperforms major foundation models on benchmarks.
- The system allows developers to customize routing based on their data.
- This approach addresses the complexity of managing multiple AI models.
2. The future of AI involves multiple specialized models.
๐ฅ89
20:41
The trend is moving away from a single monolithic model to a network of specialized models for different tasks.
- Smaller models can be more efficient and cost-effective.
- Different models excel in various areas, such as logic or writing.
- This shift allows for greater computational efficiency and flexibility.
3. Open sourcing enhances the AI routing ecosystem.
๐ฅ90
22:18
NotDiamond has open-sourced a model router to promote specialization and efficiency in AI applications.
- The open-source model outperforms existing options in terms of performance.
- It encourages innovation and reduces dependency on single providers.
- Open sourcing fosters a more competitive and diverse AI landscape.
4. Developers can customize routing based on specific needs.
๐ฅ88
29:11
NotDiamond allows users to define their own routing preferences and evaluate model performance.
- Users can upload their evaluation data for tailored routing.
- The system learns from user feedback to improve recommendations.
- This customization leads to better performance for specific applications.
5. NotDiamond offers low-latency model routing for applications.
๐ฅ92
29:28
The NotDiamond service provides efficient model routing with low latency, especially when deployed locally, achieving around 50 milliseconds for inference speed.
- API calls may introduce additional latency, typically around 400-500 milliseconds.
- The service can route queries to faster models, further improving response times.
- Local deployment allows for seamless integration into existing infrastructures.
6. Integrating NotDiamond into existing applications is straightforward.
๐ฅ90
30:38
Developers can easily integrate NotDiamond by modifying just a couple of lines of code in their existing applications using OpenAI API.
- The integration involves specifying a list of models instead of a single string.
- NotDiamond acts as a recommendation layer before sending requests to the language models.
- This approach does not require significant changes to the existing application pipeline.
7. NotDiamond supports a wide range of models and rapid onboarding.
๐ฅ88
32:41
Currently, NotDiamond supports around 40 models and can onboard new ones within 24 to 48 hours based on user requests.
- The onboarding process is efficient, allowing for quick integration of requested models.
- User feedback plays a crucial role in determining which models to prioritize for integration.
- The service aims to align model offerings with developer needs rather than just research trends.
8. Model performance varies based on specific use cases.
๐ฅ85
35:01
Different models excel in various tasks, with user feedback helping to refine routing decisions for optimal performance.
- For example, certain models perform better for JavaScript tasks, while others excel in Python.
- User feedback is essential for improving routing accuracy and model recommendations.
- Not all models are suitable for every question, and using them judiciously can save costs.
9. Leveraging LMS for customer support and HR tasks is effective.
๐ฅ88
45:27
Learning Management Systems (LMS) can significantly enhance customer support and HR tasks, although some complex queries still require human intervention.
- Certain tasks depend on implicit institutional knowledge that LMS cannot handle alone.
- Identifying when to route queries to humans is crucial for effective use.
- Training custom routers can improve performance in narrow domains.
10. Incremental improvements in LMS can yield substantial benefits.
๐ฅ92
47:41
Even small performance gains in LMS can lead to significant quality improvements, especially for large-scale applications.
- Companies can achieve better outcomes by optimizing routing between models.
- Incremental gains can exponentially enhance the quality of outputs in recursive LMS usage.
- High-scale use cases make these improvements extremely valuable.
11. Cost reduction while improving quality is achievable with LMS.
๐ฅ95
48:40
Users have reported significant quality improvements while simultaneously reducing costs, showcasing the efficiency of LMS.
- One user increased output quality by 72% while reducing costs by 14%.
- Another case showed a 30% cost reduction with slight accuracy improvements.
- This dual benefit is a key advantage of the LMS paradigm.
12. Routing models can optimize performance across multiple languages.
๐ฅ87
56:03
The LMS supports multiple languages, allowing for optimized routing based on language-specific model performance.
- Routing is not just based on language but also on the context of the query.
- Different models may excel in various languages and contexts, enhancing overall effectiveness.
- This capability broadens the usability of LMS in diverse applications.
13. Custom routing can significantly enhance model performance.
๐ฅ92
1:01:05
By training a custom router on specific workflows, users can improve output quality by 72% while reducing costs by 14%.
- This approach outperformed the previously best model, Sonet, in their specific domain.
- Routing between multiple models allows for nuanced handling of tasks like rewriting.
- The method emphasizes the importance of tailored solutions over generic models.
14. The name 'NotDiamond' reflects philosophical concepts.
๐ฅ88
1:02:20
NotDiamond references modal logic, indicating the limits of expression and the evolution of communication technology.
- The term 'not possible' highlights the boundaries of what can be expressed.
- Philosopher Ludwig Wittgenstein's quote emphasizes the constraints of language.
- The name also has aesthetic appeal, contributing to its choice.
15. NotDiamond offers free queries to users.
๐ฅ85
1:04:04
Users can access 100,000 free queries per month, with a minimal cost thereafter.
- This model encourages experimentation and usage of the platform.
- The pricing structure is designed to be accessible for users.
- Engagement through free access can foster a larger user base.