People think there's a one-size-fits-all form of "offline experiments" in search, recsys, RAG, etc. Really your job is as much to think about the methodology during _every_ experiment. Not just for accuracy, but the entire cost-benefit tradeoff equation. Off the top of my head, these are all very different things: * Getting a quick sense of a change before doing an A/B test - you can literally just gut-check the change hits the queries you expect * Training / eval a ranking model - where "NDCG" truly has to be rock solid * Debugging precision / recall - ie 'why is search behaving this way' - an offline metric (ie NDCG) is a rough guide but just one tool in your debugging arsenal * Opportunity analysis+planning - you just want a quick idea of whether signal exists, erring to potential, not accuracy * Leaderboard / competition - methodology is more-or-less "done for you" and you're optimizing in one direction (often ignoring many other factors) * Improving a system where NO A/B test exists - using labels can help you guide + debug -- maybe even make a leaderboard -- but side-by-side qualitative analysis with your team is also really big Importantly every methodology can have severe limitations but very specific benefits to where its best suited
Doug Turnbull’s Post
More Relevant Posts
-
Structuring tests is really important for readability. Three sections: 1. Arrange Set up the subject under test for testing. 2. Act Call the subject under test. 3. Assert Assert the desired behaviour. Every test should only have two blank lines - between 1 and 2, 2 and 3. Once you do this, tests become cleaner and clearer. It also exposes situations where you're setting up expectations on mocks. And that's showing how difficult the test is to read. Do your future self a favour and write tests with this clear structure. What techniques do you use to write clean tests?
To view or add a comment, sign in
-
CEO & Co-founder at Exercism · Equalising education through technology · Expert software developer · Medical AI patent-holder
😱 Find *recursion* confusing or even mystifying? Not even tried to learn it because it's too scary? We've made this video for you! Most things are easier when you approach them practically rather than theoretically, and that's what we've done in this video! We write some imperative ("normal") code to solve an exercise and then create a functional, recursive version and explore the different patterns they use. We then dive into looking at "proper" functional languages like Elixir and ML and see how those make life even easier. And at the end, Erik gives us a bit of a tour through Tail-call optimisation - a more advanced topic, but a very interesting one! I spent over 10 hours filming and editing this video because it's a really hard subject to explain. I hope that you enjoy it and the effort was worth it! https://lnkd.in/eUyxUD2Q
Recursion: A Practical Introduction (via a deep dive into List Ops!)
https://www.youtube.com/
To view or add a comment, sign in
-
💡💡 Auto-generate long-document QA for your LLMs using AttenWalker💡💡 The contributions of AttenWalker are as follow: ✅ A novel method of constructing QA for long documents, consisting of 3 modules: Span Collector, Span Linker and Answer Aggregator ✅ The Span Collector uses constituent parsing and pretrained T5's reconstruction loss to select candidate text spans ✅ The Span Linker uses pre-trained/ fine-tuned LED to link spans together through attention weight ✅ The Answer Aggregator constructs answers from linked spans using pre-trained BART ✅ Yields competitive results against existing unsupervised methods 📰 Paper: https://lnkd.in/gJv4PA97 💻 Code: https://lnkd.in/gbvGHxiq
To view or add a comment, sign in
-
-
.NET developer | ASP.NET | Docker | SQL Server | MySql | Git | VueJs | PHP | JavaScript | Linux | SEO 🇧🇷
New day, new knowledge. Let's talk about the Bubble Sort algorithm. It offers a simple and straightforward method to organize your lists effectively. Bubble Sort is a basic and intuitive algorithm for sorting lists. Its approach involves traversing the list, examining each pair of adjacent elements, and swapping them if they are in the wrong order. If you're interested in learning more about the Bubble Sort algorithm, simply click below: https://lnkd.in/dVpnCkrA
To view or add a comment, sign in
-
-
** read the problem at the bottom and try it out before reading the rest For this problem, it’s good to keep in mind that when you have a torch being taken, it must also come back. Also, note that once a person has crossed it doesn’t mean that they can’t come back hypothetically. To solve this problem, I would use a greedy algorithmic approach. In problems seeking the fastest time, this approach uses the biggest resources available to solve the problem. It’s reasonable that the faster people should go first. (2 and 1) and send back 1. From there, 10 and 5 should go, and send back 2. Lastly, 2 and 1 should go. That is 2 + 1 + 10 + 2 + 2 = 17 minutes. The overall approach is to take the minimum value(s) at each point, except the 2nd time going to, in which case you should take the max. Pseudo code: Dict = {A: 10, B: 5, C: 2, D: 1} min(A, B, C, D) = C, D (2 min) min(C,D) = D(1 min) max(A,B, D) = A, B (10 min) min(A,B,C) = C(2 min) min(C,D) = C,D(2 min) Taking a greedy approach is better if you want near-optimal solutions, but due to the small size of the problem space, we can assume this is an optimal solution. You could verify by calculating the permutations(Here's the code) from itertools import permutations times = [1, 2, 5, 10] # Generate all permutations crossing_orders = permutations(times) # Initialize variables shortest_time = float('inf') best_order = None # Iterate through each permutation to calculate the total time for order in crossing_orders: total_time = max(sum(order[:2]), order[0]) + max(order[2:]) if total_time < shortest_time: shortest_time = total_time best_order = order
To view or add a comment, sign in
-
-
If you're at least half as excited about upcoming Turbo 8 morphing functionality as I am then I am twice as excited as you are. Either way, I found it very interesting to dig into how it works, sharing in case somebody could enjoy it no less than half as much as I did writing it: https://lnkd.in/dFBCtk2t
To view or add a comment, sign in
-
MERN Stack Web Developer | React.js | Node.js | Express.js | MongoDB | Transforming Ideas into Interactive Web Applications
Here is a small Advice from me: If you are trying to build an algorithm or trying to solve a problem, try to plot your problem statement visually(in a drawing or graph form), this will help you understand the problem statement clearly and will help you craft the solution efficiently within no time. The drawing you see here is a search algorithm for a project that I am currently working on, this algorithm took me more than 2 hours to build. Before drawing the problem I tried to solve the problem for almost two hours but I was not able to do so, after when I drew the problem, it hardly took me 20 minutes to craft the solution to this problem
To view or add a comment, sign in
-
-
Passionate about inclusivity, sustainability, cycling and health. Software problems solver & author. He/him
My process for preparing a talk: 1. Open new slide deck 2. Start typing thoughts 3. Start writing code for prezzo 4. Realize something in code fails 5. Debug debug debug 6. Find bugs, open new issues 7. Spend hours trying to fix bug 8. Finally get something working hours after I started 9. Return to slide deck to realize I’m only 1 slide in 😂 10. Panic
To view or add a comment, sign in
-
It was my algorithm course project to find a way to solve the maze where the first value is the starting point and the destination point is the last value of the matrix. The user has to give an N*N matrix as an input. Future Scope- ➢ The whole project only works for a N*N matrix, that is, if the maze has a N*N matrix, it will be able to find out a path to it using this code. ➢ The project assumes that the starting point of the maze will always be at the upper left most index, which is (0,0), and the ending point of the maze will always be at the lower right most index, which is (N,N) . ➢ If the input does not have a proper path or is wrong at any single index, the output of the code will give a random or garbage value in the matrix. ➢N must be smaller than 10.
To view or add a comment, sign in
-