Please watch this video. I think this is the biggest capability that we launched in the last 18 months that people underestimate. This is the essence of what we call Data Intelligence, i.e. AI that understands *your* data.
Databricks Product Advisory Board Member & Databricks Beacon | Architect @ Kythera Labs | Technical Advisor to SunnyData
Databricks' AI Assistant went GA today. But is it truly GA ready? I took it for a SQL spin (see video), and I am happy to say that not only is it ready, but it has come a long way to become an "A" grade feature in Databricks. Here are some thoughts: Pros: +Leveraged metadata from the UC catalog to not only understand which fields to choose, but also to locate the table I was looking for without specifiying the table (see "Test 1" as an example). +While I didn't need to fix a broken query, it consistently gave functional SQL output and was able to adjust an existing query (second part of Test 6) successfully. +It was able to formulate a complex SQL query extremely well. I was 100% expecting "Test 4" to fail, but at a quick glance of the SQL query, it looked right and about what I would expect the query to look like for this finance/accounting exercise. +Fairly nit-picky prompts (Test 5 and 6) returned solid SQL code. +It followed my instructions very well without adding unnecessary details that I did not ask for. Cons: -While I didn't encounter errors during my testing, diagnosing broken code still is hit or miss from time to time. Closing Thoughts: Benjamin Rogojan (Seattle Data Guy) and I were chatting today about Text-2-SQL experiences from the past as we were recording a video, and after seeing Databricks AI Assistant go GA, I knew I had to put it through a test like this. Based on my experience with it, I would definitely pick the Databricks AI Assistant to write SQL code for me any day vs leveraging another tool such as ChatGPT, specially with its connection to your Unity Catalog metadata. Thank you for watching, and thank you to SunnyData for letting me borrow the compute!
Ali Ghodsi it is amazing to see the text-to-sql capability of the AI Assistant. It would be worthwhile to consider taking it beyond code generation to directly answer questions relevant to the business user via existing solution queries. This will give you the much needed guardrails to avoid hallucinations.
This is very impressive in asking the "what". I would love to see examples of having the "why" solved e,g, Has our sales revenue changed significantly in the last week, and why? However advanced the AI gets, any organisation that does not pay attention to its schemas, descriptive metadata and data quality will not yield these benefits, or get a bunch of erroneous answers if they attempt to leverage it. Get organised, define your patterns and follow them. That's up to the people! AI can accelerate this e.g. Databricks Gen AI table and column descriptions which do require supervision.
Wilder Bermudez, Marco Di Cio, this is the demo that we need. Simple and perfect.
Love to see this in action being tested by the many! AI/BI: Always Inventing/ Blazing Innovations Go Databricks!!
This is super helpful to understand how AI can be leveraged through basic consumers. There are three things: 1. Context size -- how big is the dataset/lake that AI operating when compiling the query? Does it produce inaccuracies when the context gets larger? 2. Internal nomenclature -- can it distinguish the internal lingo of a company relative to layman's keywords? Some companies have key dimensions they look at and may call it differently than others. 3. With points 2/3, what are the diminishing returns of accuracy and what can users look out for to make AI better? Great demo overall by Databricks, @ali, and Josue A. Bogran 🎉 Thank you for sharing!
True, I have been using Databricks Assistant for some months now, and it has definitely helped me speed up my foundational work. Here, I have shared some of the tried-and-tested prompts that will help any developer in their day-to-day work. 10 Prompts for Databricks Assistant https://www.linkedin.com/pulse/10-prompts-databricks-assistant-tasneem-hyder-cnvkc
Management Consultant, Executive MBA, Senior Enterprise Architect, Data Science, Digital Strategy and Innovation, Product Management
3wA great feature ! Risk : One generation in the future, no one will know SQL anymore since they won't need it as they rely on the text-to-SQL assistant. And if the AI assistant fails, then ... 😉