I think the people that can leverage AI to do whatever their day-to-day workflows are... We'll give people interviews where we say, 'Use ChatGPT and Kodak. Use Claude code. Use whatever tool cursor and whatever tools are available to build a website and let's see what product you're able to build in an hour.'
Brendan Foody
CEO and Co-founder, Mercor
7 quotes across 1 episode
Why experts writing AI evals is creating the fastest-growing companies in history
It's some combination of being stubborn with respect to your thesis around how the world will change, but also very open-minded with respect to exactly what form that takes and how the market's developing.
If the model is the product, then the eval is the product requirement document.
Sometimes you just need to sheer it from the market and know that it's there, the poll, to know the right places to focus. Because if it's difficult to sell, if it's extremely difficult to sell the marginal customer, you're not going to be able to grow a huge business.
You don't want to fight against them using the models. It's similar to when the calculator came out, you don't want to give people all of this arithmetic work of how do you get them to do it and not use the calculator. You want to tell them, 'Use the tools and let's see what you can do.'
The wealthiest companies in the world are willing to spend whatever it takes to improve model capabilities.
Models are only as good as their evals.