גלו את כוח העל שלכם כמנהלי מוצר: סוד ה-AI קו-פיילוט שאתם חייבים להכיר!

Hi everyone, I'm Ran Arez and this is Hi everyone, I'm Ran Arez and this is the podcast where we talk to different the podcast where we talk to different the podcast where we talk to different product managers about the problems product managers about the problems product managers about the problems they're facing, how they solved it, and they're facing, how they solved it, and they're facing, how they solved it, and what they learned along the way. And what they learned along the way. And what they learned along the way. And today we have a very important podcast today we have a very important podcast today we have a very important podcast episode. Today we are hosting Ben Arez episode. Today we are hosting Ben Arez episode. Today we are hosting Ben Arez to talk about an AI co-pilot experience to talk about an AI co-pilot experience to talk about an AI co-pilot experience and he's going to share with us his and he's going to share with us his and he's going to share with us his challenges and his learning and his challenges and his learning and his challenges and his learning and his trade-offs. So Ben, thank you so much trade-offs. So Ben, thank you so much trade-offs. So Ben, thank you so much for uh being here. for uh being here. for uh being here. >> Yeah, thanks for having me. This is fun. >> Yeah, thanks for having me. This is fun. >> Yeah, thanks for having me. This is fun. >> Amazing. So, for everyone who may not >> Amazing. So, for everyone who may not >> Amazing. So, for everyone who may not know you, maybe just introduce yourself know you, maybe just introduce yourself know you, maybe just introduce yourself a bit. a bit. a bit. >> Sure. Uh, so yeah, my name is Ben. I >> Sure. Uh, so yeah, my name is Ben. I >> Sure. Uh, so yeah, my name is Ben. I live in uh Brooklyn, New York with my my live in uh Brooklyn, New York with my my live in uh Brooklyn, New York with my my wife and our three-year-old daughter and wife and our three-year-old daughter and wife and our three-year-old daughter and uh 8-year-old golden retriever. Uh, we uh 8-year-old golden retriever. Uh, we uh 8-year-old golden retriever. Uh, we moved here a little over four years ago moved here a little over four years ago moved here a little over four years ago after 8 years in San Francisco. And after 8 years in San Francisco. And after 8 years in San Francisco. And before that, uh, grew up between the US before that, uh, grew up between the US before that, uh, grew up between the US and Israel. Went to school in Florida. and Israel. Went to school in Florida. and Israel. Went to school in Florida. Um, yeah, I started my career in Um, yeah, I started my career in Um, yeah, I started my career in finance. I got out very early and then I finance. I got out very early and then I finance. I got out very early and then I was a failed founder, but learned a lot was a failed founder, but learned a lot was a failed founder, but learned a lot and found my way into product and found my way into product and found my way into product management. Uh, joined a company called management. Uh, joined a company called management. Uh, joined a company called Life 360. uh as their 35th employee Life 360. uh as their 35th employee Life 360. uh as their 35th employee hired by another Israeli uh product guy hired by another Israeli uh product guy hired by another Israeli uh product guy named Idomar Novik and then discovered named Idomar Novik and then discovered named Idomar Novik and then discovered product management there and I'd say the product management there and I'd say the product management there and I'd say the rest is uh kind of history. I ended up rest is uh kind of history. I ended up rest is uh kind of history. I ended up spending uh the last the the following spending uh the last the the following spending uh the last the the following 10 years across uh series A B C 10 years across uh series A B C 10 years across uh series A B C startups. I also uh was at Facebook for startups. I also uh was at Facebook for startups. I also uh was at Facebook for a period which kind of like one of the a period which kind of like one of the a period which kind of like one of the most influential experiences that led to most influential experiences that led to most influential experiences that led to what we're going to talk about today. what we're going to talk about today. what we're going to talk about today. Um, and uh, I'd consider myself at my at Um, and uh, I'd consider myself at my at Um, and uh, I'd consider myself at my at my core like a zero to one product my core like a zero to one product my core like a zero to one product person. And a lot of the work that I've person. And a lot of the work that I've person. And a lot of the work that I've done in larger companies, um, you know, done in larger companies, um, you know, done in larger companies, um, you know, while while potentially more impactful while while potentially more impactful while while potentially more impactful on the on the scoreboard, I think still, on the on the scoreboard, I think still, on the on the scoreboard, I think still, uh, in my heart never I've always just uh, in my heart never I've always just uh, in my heart never I've always just loved being super hands-on with building loved being super hands-on with building loved being super hands-on with building things, talking to users, and just things, talking to users, and just things, talking to users, and just creating creating new things.

creating creating new things. creating creating new things. >> Amazing. So, let's before we dive in, >> Amazing. So, let's before we dive in, >> Amazing. So, let's before we dive in, let's just set the stage. What are we let's just set the stage. What are we let's just set the stage. What are we going to talk about today? Why is it going to talk about today? Why is it going to talk about today? Why is it important to other product managers? important to other product managers? important to other product managers? Yeah. So, I kind of alluded this to this Yeah. So, I kind of alluded this to this Yeah. So, I kind of alluded this to this in the intro, but the last couple years in the intro, but the last couple years in the intro, but the last couple years have been uh kind of like an entirely have been uh kind of like an entirely have been uh kind of like an entirely different chapter of my career. So, different chapter of my career. So, different chapter of my career. So, until two years ago, I was in full-time until two years ago, I was in full-time until two years ago, I was in full-time PM work. As I mentioned, I was the first PM work. As I mentioned, I was the first PM work. As I mentioned, I was the first PM at three different startups. I was a PM at three different startups. I was a PM at three different startups. I was a PM at Facebook, attentive, um I was a PM at Facebook, attentive, um I was a PM at Facebook, attentive, um I was a founder. And then two years ago, um I founder. And then two years ago, um I founder. And then two years ago, um I decided to do something different uh decided to do something different uh decided to do something different uh with my career and I kind of went off with my career and I kind of went off with my career and I kind of went off off the the beaten trail and started um off the the beaten trail and started um off the the beaten trail and started um what I might consider as a what I might consider as a what I might consider as a solarreneurship journey. Started off solarreneurship journey. Started off solarreneurship journey. Started off thinking about doing fractional product thinking about doing fractional product thinking about doing fractional product work like advising, consulting uh and I work like advising, consulting uh and I work like advising, consulting uh and I did I have done some of that over the did I have done some of that over the did I have done some of that over the last couple years but I would say the last couple years but I would say the last couple years but I would say the thing that really pulled me uh more than thing that really pulled me uh more than thing that really pulled me uh more than anything else that I've done over the anything else that I've done over the anything else that I've done over the last two years is uh product management last two years is uh product management last two years is uh product management interview preparation. Um I used to do interview preparation. Um I used to do interview preparation. Um I used to do this as a favor to people to help them this as a favor to people to help them this as a favor to people to help them get ready for interviews at places like get ready for interviews at places like get ready for interviews at places like Meta. And then at some point I helped Meta. And then at some point I helped Meta. And then at some point I helped someone land a very lucrative um offer someone land a very lucrative um offer someone land a very lucrative um offer after spending about six hours helping after spending about six hours helping after spending about six hours helping them prepare. Wow. them prepare. Wow. them prepare. Wow. >> And they're like, "You're an idiot for >> And they're like, "You're an idiot for >> And they're like, "You're an idiot for not charging money for this." So um I not charging money for this." So um I not charging money for this." So um I ended up starting to charge and um and ended up starting to charge and um and ended up starting to charge and um and then that all led to a Maven course that then that all led to a Maven course that then that all led to a Maven course that I launched about a year and a half ago, I launched about a year and a half ago, I launched about a year and a half ago, coming up on I guess like two years in coming up on I guess like two years in coming up on I guess like two years in in April. in April. in April. >> Wow. Um, and that Maven course paved the

>> Wow. Um, and that Maven course paved the >> Wow. Um, and that Maven course paved the path to my co-pilot, which is what we're path to my co-pilot, which is what we're path to my co-pilot, which is what we're going to talk about today, as a tool for going to talk about today, as a tool for going to talk about today, as a tool for helping my students, but ultimately that helping my students, but ultimately that helping my students, but ultimately that the impact of that expanded well outside the impact of that expanded well outside the impact of that expanded well outside of my course. And my niche, uh, I would of my course. And my niche, uh, I would of my course. And my niche, uh, I would say has really become product sense and say has really become product sense and say has really become product sense and analytical thinking interview analytical thinking interview analytical thinking interview preparation like the ones that are used preparation like the ones that are used preparation like the ones that are used at companies like Meta. I think you guys at companies like Meta. I think you guys at companies like Meta. I think you guys at Monday um, we're doing that as well. at Monday um, we're doing that as well. at Monday um, we're doing that as well. Lots of other companies use those Lots of other companies use those Lots of other companies use those interviews, too. And then um I have a interviews, too. And then um I have a interviews, too. And then um I have a few other things I'm spending my time on few other things I'm spending my time on few other things I'm spending my time on these days, but I think you know it's these days, but I think you know it's these days, but I think you know it's all it's all in the general PM interview all it's all in the general PM interview all it's all in the general PM interview prep space. prep space. prep space. >> Yeah. Great. So you said throughout the >> Yeah. Great. So you said throughout the >> Yeah. Great. So you said throughout the lines like co-pilot. How do you define a lines like co-pilot. How do you define a lines like co-pilot. How do you define a co-pilot? co-pilot? co-pilot? >> Yeah. I mean I think it's kind of um >> Yeah. I mean I think it's kind of um >> Yeah. I mean I think it's kind of um it's one of those words that has a lot it's one of those words that has a lot it's one of those words that has a lot of different meanings to a lot of of different meanings to a lot of of different meanings to a lot of different people kind of like agent. Um different people kind of like agent. Um different people kind of like agent. Um so I would say so I would say so I would say >> um let's just talk about what talk about >> um let's just talk about what talk about >> um let's just talk about what talk about it as a tool it as a tool it as a tool >> and then you know I can talk to you I >> and then you know I can talk to you I >> and then you know I can talk to you I can tell you why I decided to call it a can tell you why I decided to call it a can tell you why I decided to call it a co-pilot but but what the tool does at co-pilot but but what the tool does at co-pilot but but what the tool does at its core is you could think of people um its core is you could think of people um its core is you could think of people um using claude chatpt Gemini whichever LLM using claude chatpt Gemini whichever LLM using claude chatpt Gemini whichever LLM is their tool of choice and a couple is their tool of choice and a couple is their tool of choice and a couple years ago um actually you know what it years ago um actually you know what it years ago um actually you know what it was probably just a year and a half ago was probably just a year and a half ago was probably just a year and a half ago or so we had Tal Raviv on on my podcast or so we had Tal Raviv on on my podcast or so we had Tal Raviv on on my podcast and I got to know Tal and I got to know Tal and I got to know Tal >> and Tal was kind of just like telling >> and Tal was kind of just like telling >> and Tal was kind of just like telling everyone about Claude Projects and I'd everyone about Claude Projects and I'd everyone about Claude Projects and I'd never played with Claude Projects never played with Claude Projects never played with Claude Projects before. Um, before. Um, before. Um, >> but I started playing with it and I had >> but I started playing with it and I had >> but I started playing with it and I had a light bulb moment go off and what the a light bulb moment go off and what the a light bulb moment go off and what the light bulb moment was is I had at that light bulb moment was is I had at that light bulb moment was is I had at that point been teaching my course for about point been teaching my course for about point been teaching my course for about five or six months and I started to get five or six months and I started to get five or six months and I started to get a really good feel for what people were a really good feel for what people were a really good feel for what people were asking me about in my course. They were asking me about in my course. They were asking me about in my course. They were asking me for help filling out my asking me for help filling out my asking me for help filling out my templates. I did some um I created these templates. I did some um I created these templates. I did some um I created these very popular and they're free like very popular and they're free like very popular and they're free like interview templates for product sense interview templates for product sense interview templates for product sense and analytical thinking interviews and and analytical thinking interviews and and analytical thinking interviews and people would make a copy and start people would make a copy and start people would make a copy and start working through a question from a working through a question from a working through a question from a question bank and then they would fill question bank and then they would fill question bank and then they would fill it out and they would ask me hey like it out and they would ask me hey like it out and they would ask me hey like could you give me feedback on how this could you give me feedback on how this could you give me feedback on how this submission is um and that would be very submission is um and that would be very submission is um and that would be very timeconuming because it wasn't like a timeconuming because it wasn't like a timeconuming because it wasn't like a 10-second thing. It was like I had to 10-second thing. It was like I had to 10-second thing. It was like I had to understand the whole context of the understand the whole context of the understand the whole context of the exercise. So I was like I need I need a exercise. So I was like I need I need a exercise. So I was like I need I need a way to scale my ability to give feedback way to scale my ability to give feedback way to scale my ability to give feedback on those filled out submissions. And on those filled out submissions. And on those filled out submissions. And then the other problem was some people then the other problem was some people then the other problem was some people get very fixated on the way a question get very fixated on the way a question get very fixated on the way a question gets asked in the question bank. So like gets asked in the question bank. So like gets asked in the question bank. So like they see a a flavor of a of a way that they see a a flavor of a of a way that they see a a flavor of a of a way that an interviewer asked a question they've an interviewer asked a question they've an interviewer asked a question they've never seen before and it kind of like never seen before and it kind of like never seen before and it kind of like can be kind of scary for them. can be kind of scary for them. can be kind of scary for them. >> So they're like how would you tackle >> So they're like how would you tackle >> So they're like how would you tackle this? Like how does this come back into this? Like how does this come back into this? Like how does this come back into your framework? So I found myself your framework? So I found myself your framework? So I found myself spending a lot of time help you know spending a lot of time help you know spending a lot of time help you know getting on calls with people and being getting on calls with people and being getting on calls with people and being like here's exactly how you kind of fit like here's exactly how you kind of fit like here's exactly how you kind of fit this into my framework. Don't be this into my framework. Don't be this into my framework. Don't be intimidated by the question. So I had intimidated by the question. So I had intimidated by the question. So I had these two big problems and I was just these two big problems and I was just these two big problems and I was just spending like way too much time with spending like way too much time with spending like way too much time with students trying to answer that and I students trying to answer that and I students trying to answer that and I knew that wasn't going to scale. So what knew that wasn't going to scale. So what knew that wasn't going to scale. So what I started tinkering with in Clawude I started tinkering with in Clawude I started tinkering with in Clawude projects is a way to basically codify my projects is a way to basically codify my projects is a way to basically codify my framework for how to answer these framework for how to answer these framework for how to answer these questions. Give it my template so it's questions. Give it my template so it's questions. Give it my template so it's aware of how the template works and then aware of how the template works and then aware of how the template works and then give it some examples of how filled out give it some examples of how filled out give it some examples of how filled out templates might look or how filled out templates might look or how filled out templates might look or how filled out submissions might work and then give it submissions might work and then give it submissions might work and then give it a very clear evaluation rubric to start. a very clear evaluation rubric to start. a very clear evaluation rubric to start. And then I just like that was kind of And then I just like that was kind of And then I just like that was kind of like the in that was like my light bulb like the in that was like my light bulb like the in that was like my light bulb moment for the co-pilot. I end up moment for the co-pilot. I end up moment for the co-pilot. I end up calling it the co-pilot because what it calling it the co-pilot because what it calling it the co-pilot because what it is is it's it's like a practice co-pilot is is it's it's like a practice co-pilot is is it's it's like a practice co-pilot when I tell people you're going to spend when I tell people you're going to spend when I tell people you're going to spend 20 30 hours maybe getting ready for your 20 30 hours maybe getting ready for your 20 30 hours maybe getting ready for your meta interviews or you know your product meta interviews or you know your product meta interviews or you know your product sense analytical thinking interviews at sense analytical thinking interviews at sense analytical thinking interviews at another company. Um how are you going to another company. Um how are you going to another company. Um how are you going to use that time? Are you going to spend it use that time? Are you going to spend it use that time? Are you going to spend it just doing work on your own? Are you just doing work on your own? Are you just doing work on your own? Are you going to spend that all on mocks? Are going to spend that all on mocks? Are going to spend that all on mocks? Are you going to spend it doing something you going to spend it doing something you going to spend it doing something else? Like what are you going to use else? Like what are you going to use else? Like what are you going to use that time for? So, I tried to create a that time for? So, I tried to create a that time for? So, I tried to create a very opinionated path for how people very opinionated path for how people very opinionated path for how people could spend that time. And I generally could spend that time. And I generally could spend that time. And I generally tell people to start with a co-pilot tell people to start with a co-pilot tell people to start with a co-pilot when they're ready to get reps with the when they're ready to get reps with the when they're ready to get reps with the material, but before they do mock material, but before they do mock material, but before they do mock interviews. interviews. interviews. >> I think when we think about other PMs >> I think when we think about other PMs >> I think when we think about other PMs listening to this is we try to frame it listening to this is we try to frame it listening to this is we try to frame it as either a challenge of how do you as either a challenge of how do you as either a challenge of how do you scale yourself and how do you scale yourself and how do you scale yourself and how do you personalize your answers when you're not personalize your answers when you're not personalize your answers when you're not there. And when we think about this there. And when we think about this there. And when we think about this framework of personalization and scale,

framework of personalization and scale, framework of personalization and scale, I think this is the initial questions I think this is the initial questions I think this is the initial questions when we need to ask when we could think when we need to ask when we could think when we need to ask when we could think about maybe a co-pilot is right for me about maybe a co-pilot is right for me about maybe a co-pilot is right for me at the situation. I think today what at the situation. I think today what at the situation. I think today what we're going to do is we're going to we're going to do is we're going to we're going to do is we're going to break down both through your use case break down both through your use case break down both through your use case but also to expand it to other let's but also to expand it to other let's but also to expand it to other let's call it Ben's playbook for a co-pilot to call it Ben's playbook for a co-pilot to call it Ben's playbook for a co-pilot to really understand the nitty-gritty and really understand the nitty-gritty and really understand the nitty-gritty and everything that we need to think about everything that we need to think about everything that we need to think about when we think about launching a co-pilot when we think about launching a co-pilot when we think about launching a co-pilot to either scale ourselves, personalize to either scale ourselves, personalize to either scale ourselves, personalize ourel or both of them. Okay, sounds ourel or both of them. Okay, sounds ourel or both of them. Okay, sounds good. Yeah, let's do it. good. Yeah, let's do it. good. Yeah, let's do it. >> Okay. So, let let's take your case study >> Okay. So, let let's take your case study >> Okay. So, let let's take your case study and and and let's let's start. You said and and and let's let's start. You said and and and let's let's start. You said it was about a year ago when you started it was about a year ago when you started it was about a year ago when you started it. Maybe walk us through the steps you it. Maybe walk us through the steps you it. Maybe walk us through the steps you took to to get it off the ground. took to to get it off the ground. took to to get it off the ground. >> Yeah. I would say the very first thing

>> Yeah. I would say the very first thing >> Yeah. I would say the very first thing was just figuring out if it even works, was just figuring out if it even works, was just figuring out if it even works, >> right? So, I wasn't thinking about >> right? So, I wasn't thinking about >> right? So, I wasn't thinking about marketing. I wasn't thinking about marketing. I wasn't thinking about marketing. I wasn't thinking about packaging or pricing. Um the original packaging or pricing. Um the original packaging or pricing. Um the original idea was I was wondering if I could idea was I was wondering if I could idea was I was wondering if I could build something that helped me um do build something that helped me um do build something that helped me um do something as like an internal tool as an something as like an internal tool as an something as like an internal tool as an instructor. instructor. instructor. >> Okay. So deep understanding of my own >> Okay. So deep understanding of my own >> Okay. So deep understanding of my own workflow um was kind of like the workflow um was kind of like the workflow um was kind of like the foundation and obviously having a deep foundation and obviously having a deep foundation and obviously having a deep set of like pretty strong opinions about set of like pretty strong opinions about set of like pretty strong opinions about what good and like not good looks like. what good and like not good looks like. what good and like not good looks like. >> Um >> Um >> Um I'm we're going to be trying to I'm we're going to be trying to I'm we're going to be trying to generalize this live for the first time. generalize this live for the first time. generalize this live for the first time. Okay. So I I think some of this is going Okay. So I I think some of this is going Okay. So I I think some of this is going to be a bit you know raw as we talk to be a bit you know raw as we talk to be a bit you know raw as we talk about it. But maybe if the audience has about it. But maybe if the audience has about it. But maybe if the audience has any um any um any um uh kind of like light bulb moments as we uh kind of like light bulb moments as we uh kind of like light bulb moments as we talk through this for them learning how talk through this for them learning how talk through this for them learning how to apply this to their life, I actually to apply this to their life, I actually to apply this to their life, I actually would really appreciate hearing that would really appreciate hearing that would really appreciate hearing that from them because I'm I'm on the lookout from them because I'm I'm on the lookout from them because I'm I'm on the lookout for more ways that people are doing this for more ways that people are doing this for more ways that people are doing this kind of stuff because I don't think kind of stuff because I don't think kind of stuff because I don't think there's anything particularly special there's anything particularly special there's anything particularly special about me. But I don't think I've also about me. But I don't think I've also about me. But I don't think I've also codified this into like a clear playbook codified this into like a clear playbook codified this into like a clear playbook that just like anyone can do that just like anyone can do that just like anyone can do immediately. immediately. immediately. >> Yeah. So it's a work in progress. That's >> Yeah. So it's a work in progress. That's >> Yeah. So it's a work in progress. That's okay. I think okay. I think okay. I think >> yeah it is a work in progress and I >> yeah it is a work in progress and I >> yeah it is a work in progress and I would say like so that that was kind of would say like so that that was kind of would say like so that that was kind of like the first moment is like trying to like the first moment is like trying to like the first moment is like trying to see if I could work through a single see if I could work through a single see if I could work through a single question with my with this project and I question with my with this project and I question with my with this project and I started with a product sense question. started with a product sense question. started with a product sense question. >> Uh in hindsight I probably would have >> Uh in hindsight I probably would have >> Uh in hindsight I probably would have started with analytical thinking. I started with analytical thinking. I started with analytical thinking. I think it's actually kind of a little think it's actually kind of a little think it's actually kind of a little easier to codify um analytical thinking easier to codify um analytical thinking easier to codify um analytical thinking interviews in this way than product interviews in this way than product interviews in this way than product sense but um but nonetheless that's the sense but um but nonetheless that's the sense but um but nonetheless that's the first step is just play with it for first step is just play with it for first step is just play with it for yourself. build something that you could yourself. build something that you could yourself. build something that you could use that adds value to you and gives you use that adds value to you and gives you use that adds value to you and gives you leverage for something that you find leverage for something that you find leverage for something that you find yourself doing a lot. yourself doing a lot. yourself doing a lot. >> Okay. Okay. >> Okay. Okay. >> Okay. Okay. >> So, and that was the point where you >> So, and that was the point where you >> So, and that was the point where you thought about at first it was just thought about at first it was just thought about at first it was just internal just for you to get more internal just for you to get more internal just for you to get more leverage as an instructor, right? leverage as an instructor, right? leverage as an instructor, right? >> So, you had the need of actually >> So, you had the need of actually >> So, you had the need of actually answering more and more people and answering more and more people and answering more and more people and that's where you started working through that's where you started working through that's where you started working through the product sensory, right? the product sensory, right? the product sensory, right? >> Correct. Um there's there was that and >> Correct. Um there's there was that and >> Correct. Um there's there was that and there was also kind of like a business there was also kind of like a business there was also kind of like a business need. You know, as like every good need. You know, as like every good need. You know, as like every good product person, you're trying to solve a product person, you're trying to solve a product person, you're trying to solve a human problem and also trying to solve a human problem and also trying to solve a human problem and also trying to solve a business problem. And the business business problem. And the business business problem. And the business problem was I couldn't see a path for problem was I couldn't see a path for problem was I couldn't see a path for myself to scale my course beyond maybe myself to scale my course beyond maybe myself to scale my course beyond maybe like 10 or 15 students at the current at like 10 or 15 students at the current at like 10 or 15 students at the current at the amount of time I was spending with the amount of time I was spending with the amount of time I was spending with my students. So I had a business need my students. So I had a business need my students. So I had a business need that's like if this is going to be able that's like if this is going to be able that's like if this is going to be able to grow, I need I need some systems that to grow, I need I need some systems that to grow, I need I need some systems that allow me to grow this. Mhm. allow me to grow this. Mhm. allow me to grow this. Mhm. >> Um, and then I think there was another >> Um, and then I think there was another >> Um, and then I think there was another business need slash a uh pricing and business need slash a uh pricing and business need slash a uh pricing and positioning need which was I wanted to positioning need which was I wanted to positioning need which was I wanted to be able to justif to command a more be able to justif to command a more be able to justif to command a more premium price for my course. And I was premium price for my course. And I was premium price for my course. And I was just like really strongly thinking about just like really strongly thinking about just like really strongly thinking about what is the most valuable thing I could what is the most valuable thing I could what is the most valuable thing I could add to the course beyond what I'm add to the course beyond what I'm add to the course beyond what I'm already doing. And some kind of tool already doing. And some kind of tool already doing. And some kind of tool that people could use to to do their own that people could use to to do their own that people could use to to do their own practice that was like a proprietary practice that was like a proprietary practice that was like a proprietary tool felt to me like it would be very tool felt to me like it would be very tool felt to me like it would be very value ad. Mhm. value ad. Mhm. value ad. Mhm. >> So when all those conditions were right, >> So when all those conditions were right, >> So when all those conditions were right, that kind of made me start really that kind of made me start really that kind of made me start really playing with this and getting excited. playing with this and getting excited. playing with this and getting excited. >> Okay. But but the first thing is I think >> Okay. But but the first thing is I think >> Okay. But but the first thing is I think first of all we need to see demand where first of all we need to see demand where first of all we need to see demand where you try to repeat yourself over and over you try to repeat yourself over and over you try to repeat yourself over and over again in a specific area and then you again in a specific area and then you again in a specific area and then you say okay I have a market pool of say okay I have a market pool of say okay I have a market pool of actually questions coming back and they actually questions coming back and they actually questions coming back and they usually are similar questions but on usually are similar questions but on usually are similar questions but on maybe slightly different ways of asking maybe slightly different ways of asking maybe slightly different ways of asking the same thing. Right? I think that's the same thing. Right? I think that's the same thing. Right? I think that's the first thing of actually seeing a the first thing of actually seeing a the first thing of actually seeing a general demand that's very similar. general demand that's very similar. general demand that's very similar. Right? It's not like questions that are Right? It's not like questions that are Right? It's not like questions that are very very different, right? very very different, right? very very different, right? >> Uh totally agree. And and and it's clear >> Uh totally agree. And and and it's clear >> Uh totally agree. And and and it's clear in hindsight that there were two buckets in hindsight that there were two buckets in hindsight that there were two buckets of questions I was getting asked, but in of questions I was getting asked, but in of questions I was getting asked, but in the moment it wasn't obvious. Like in the moment it wasn't obvious. Like in the moment it wasn't obvious. Like in the moment I just felt overwhelmed, the moment I just felt overwhelmed, the moment I just felt overwhelmed, right? right? right? >> Like I'm just like, "Oh my god, I just >> Like I'm just like, "Oh my god, I just >> Like I'm just like, "Oh my god, I just ran this course. I I ran my first cohort ran this course. I I ran my first cohort ran this course. I I ran my first cohort in July and then I took three months and in July and then I took three months and in July and then I took three months and I ran the next cohort in October and I ran the next cohort in October and I ran the next cohort in October and then it was around um the holiday then it was around um the holiday then it was around um the holiday season, like end of December. I was season, like end of December. I was season, like end of December. I was getting ready for my January cohort and getting ready for my January cohort and getting ready for my January cohort and I was kind of like, I don't know how I was kind of like, I don't know how I was kind of like, I don't know how this is going to go. I'm really worried this is going to go. I'm really worried this is going to go. I'm really worried about that, but I definitely feel about that, but I definitely feel about that, but I definitely feel overwhelmed. So, I just started kind of overwhelmed. So, I just started kind of overwhelmed. So, I just started kind of chipping away at the problem. In chipping away at the problem. In chipping away at the problem. In hindsight, hindsight, hindsight, >> it's clear to me that my attempt to chip >> it's clear to me that my attempt to chip >> it's clear to me that my attempt to chip away at the problem was away at the problem was away at the problem was >> had implicitly prioritized, let's say, a >> had implicitly prioritized, let's say, a >> had implicitly prioritized, let's say, a couple big buckets of questions that I couple big buckets of questions that I couple big buckets of questions that I got asked. got asked. got asked. >> Okay. And then so we see the demand and >> Okay. And then so we see the demand and >> Okay. And then so we see the demand and we see that we can start scaling we see that we can start scaling we see that we can start scaling ourselves ourselves ourselves >> through this. Then you started working >> through this. Then you started working >> through this. Then you started working on uh cloud projects right? on uh cloud projects right? on uh cloud projects right? >> Yeah. >> Yeah. >> Yeah. >> Okay. What was that uh looking like is >> Okay. What was that uh looking like is >> Okay. What was that uh looking like is in in in the real real time what did you in in in the real real time what did you in in in the real real time what did you do? do? do? >> Yeah. So uh the cool thing about cloud >> Yeah. So uh the cool thing about cloud >> Yeah. So uh the cool thing about cloud about I guess like AI tools at that about I guess like AI tools at that about I guess like AI tools at that point cloud projects at that point is point cloud projects at that point is point cloud projects at that point is there weren't that many um levers to there weren't that many um levers to there weren't that many um levers to play with play with play with >> right? So you had instruct you had you >> right? So you had instruct you had you >> right? So you had instruct you had you create a project and then there's like create a project and then there's like create a project and then there's like instructions and there's files. instructions and there's files. instructions and there's files. >> Okay. So >> Okay. So >> Okay. So >> that's basically all it was and it was >> that's basically all it was and it was >> that's basically all it was and it was doing rag um to kind of try to come up doing rag um to kind of try to come up doing rag um to kind of try to come up with answers. Um and so with answers. Um and so with answers. Um and so uh the first thing I tried to do is I uh the first thing I tried to do is I uh the first thing I tried to do is I just didn't put in any files. I didn't just didn't put in any files. I didn't just didn't put in any files. I didn't put in any instructions. put in any instructions. put in any instructions. I just asked an interview question. I just asked an interview question. I just asked an interview question. >> Okay. So, I said, you know, um, how >> Okay. So, I said, you know, um, how >> Okay. So, I said, you know, um, how would you I think I might have asked would you I think I might have asked would you I think I might have asked like the most common question that I've like the most common question that I've like the most common question that I've seen over the years, which is something seen over the years, which is something seen over the years, which is something like, how would you build a product for like, how would you build a product for like, how would you build a product for volunteering? volunteering? volunteering? >> Yeah. >> Yeah. >> Yeah. >> Right. I I may have said you're a PM at >> Right. I I may have said you're a PM at >> Right. I I may have said you're a PM at Meta. Um, how would you build a product Meta. Um, how would you build a product Meta. Um, how would you build a product for volunteering? Or something like for volunteering? Or something like for volunteering? Or something like that. Um, and I just kind of saw what it that. Um, and I just kind of saw what it that. Um, and I just kind of saw what it came up with and obviously it jumped came up with and obviously it jumped came up with and obviously it jumped right into the solution right into the solution right into the solution >> solution, right? >> solution, right? >> solution, right? >> Like a bad PM, right? >> Like a bad PM, right? >> Like a bad PM, right? >> Like a bad PM. So I was kind of like, >> Like a bad PM. So I was kind of like, >> Like a bad PM. So I was kind of like, okay, like I guess if I just it would okay, like I guess if I just it would okay, like I guess if I just it would have no it would have no way of knowing have no it would have no way of knowing have no it would have no way of knowing that that's not the right way to do to that that's not the right way to do to that that's not the right way to do to do this. So, what if I go in the do this. So, what if I go in the do this. So, what if I go in the instructions instructions instructions um and I say, "Hey, like every chat in um and I say, "Hey, like every chat in um and I say, "Hey, like every chat in this project is going to be me asking this project is going to be me asking this project is going to be me asking you an interview question and before you you an interview question and before you you an interview question and before you jump into solutions, I just want you to jump into solutions, I just want you to jump into solutions, I just want you to frame the context around why this frame the context around why this frame the context around why this matters and then, you know, walk me matters and then, you know, walk me matters and then, you know, walk me through who you might build this for and through who you might build this for and through who you might build this for and then tell me what their problems might then tell me what their problems might then tell me what their problems might be and then and then I want you to land be and then and then I want you to land be and then and then I want you to land on a solution at the end." Mhm. on a solution at the end." Mhm. on a solution at the end." Mhm. >> So, I just started very simple like that >> So, I just started very simple like that >> So, I just started very simple like that and then I went back to the chat and and then I went back to the chat and and then I went back to the chat and Claude Projects has this button called Claude Projects has this button called Claude Projects has this button called retry retry retry >> and you could hit retry with no changes >> and you could hit retry with no changes >> and you could hit retry with no changes and what that does is it basically has and what that does is it basically has and what that does is it basically has the the the AI take another stab at the the the AI take another stab at the the the AI take another stab at generating the response but it does pull generating the response but it does pull generating the response but it does pull the latest kind of instructions from the the latest kind of instructions from the the latest kind of instructions from the project project project >> and then I got to see what it did you >> and then I got to see what it did you >> and then I got to see what it did you know with the latest instructions and know with the latest instructions and know with the latest instructions and then I was like okay that's a little bit then I was like okay that's a little bit then I was like okay that's a little bit better but this part is off and that I better but this part is off and that I better but this part is off and that I basically just did that like a few basically just did that like a few basically just did that like a few hundred times over the course of hundred times over the course of hundred times over the course of probably a couple weeks. probably a couple weeks. probably a couple weeks. >> Mhm. >> Mhm. >> Mhm. >> Just to kind of get the output from the >> Just to kind of get the output from the >> Just to kind of get the output from the response for for product sense and response for for product sense and response for for product sense and analytical thinking to be what I would analytical thinking to be what I would analytical thinking to be what I would kind of expect to see in a in a passing

kind of expect to see in a in a passing kind of expect to see in a in a passing answer. answer. answer. >> So I want to I want to highlight two >> So I want to I want to highlight two >> So I want to I want to highlight two things that I think you did which are things that I think you did which are things that I think you did which are really really important is the first really really important is the first really really important is the first thing is seeing the baseline. What do we thing is seeing the baseline. What do we thing is seeing the baseline. What do we do without any instructions? Like you do without any instructions? Like you do without any instructions? Like you saw the baseline case and you saw that saw the baseline case and you saw that saw the baseline case and you saw that you could improve that dramatically or you could improve that dramatically or you could improve that dramatically or at least you had an hypothesis. And I at least you had an hypothesis. And I at least you had an hypothesis. And I think this is really crucial because a think this is really crucial because a think this is really crucial because a lot of times as PMs we run into lot of times as PMs we run into lot of times as PMs we run into solutions as well and we try to create solutions as well and we try to create solutions as well and we try to create more instructions where maybe the best more instructions where maybe the best more instructions where maybe the best the base case is okay. And I think when the base case is okay. And I think when the base case is okay. And I think when you saw that it runs to solutions like you saw that it runs to solutions like you saw that it runs to solutions like we do as PM sometimes I think that's we do as PM sometimes I think that's we do as PM sometimes I think that's that's a great signal to start as a that's a great signal to start as a that's a great signal to start as a baseline. The second thing is instead of baseline. The second thing is instead of baseline. The second thing is instead of creating all of the instructions at once creating all of the instructions at once creating all of the instructions at once and seeing the results you built it like and seeing the results you built it like and seeing the results you built it like bottom up. You started with the baseline bottom up. You started with the baseline bottom up. You started with the baseline and then you added one instruction and and then you added one instruction and and then you added one instruction and another one and another one and you another one and another one and you another one and another one and you actually control the the output with a actually control the the output with a actually control the the output with a lot more granularity than what we I see lot more granularity than what we I see lot more granularity than what we I see sometimes is PMs throwing a lot of sometimes is PMs throwing a lot of sometimes is PMs throwing a lot of instructions and having no idea how a instructions and having no idea how a instructions and having no idea how a word might change their results. word might change their results. word might change their results. And I think the last thing that I want And I think the last thing that I want And I think the last thing that I want to highlight is the fact that you to highlight is the fact that you to highlight is the fact that you retried but without the context. So it's retried but without the context. So it's retried but without the context. So it's like a clean environment that you like a clean environment that you like a clean environment that you started from fresh every time. So you started from fresh every time. So you started from fresh every time. So you didn't have like a dirty environment of didn't have like a dirty environment of didn't have like a dirty environment of context in the AI to actually learn from context in the AI to actually learn from context in the AI to actually learn from their mistakes and not from your their mistakes and not from your their mistakes and not from your instructions, right? instructions, right? instructions, right? >> Yeah. And then when I did feel like the >> Yeah. And then when I did feel like the >> Yeah. And then when I did feel like the environment was getting contaminated by environment was getting contaminated by environment was getting contaminated by the context of a given chat thread, I the context of a given chat thread, I the context of a given chat thread, I would just start a new chat and be like, would just start a new chat and be like, would just start a new chat and be like, "Okay, now let's start from scratch." "Okay, now let's start from scratch." "Okay, now let's start from scratch." Like it has none of that context. And I Like it has none of that context. And I Like it has none of that context. And I did just like lots and lots of threads. did just like lots and lots of threads. did just like lots and lots of threads. But yeah, you're right. I what what I But yeah, you're right. I what what I But yeah, you're right. I what what I was trying to do is I was trying to kind was trying to do is I was trying to kind was trying to do is I was trying to kind of get a feel for this machine, of get a feel for this machine, of get a feel for this machine, >> right? >> right? >> right? >> What is this thing? How does it work? >> What is this thing? How does it work? >> What is this thing? How does it work? like how um how much of this kind of like how um how much of this kind of like how um how much of this kind of like can I really nudge in the direction like can I really nudge in the direction like can I really nudge in the direction that I want to nudge it versus how much that I want to nudge it versus how much that I want to nudge it versus how much of it regardless of nudging is just not of it regardless of nudging is just not of it regardless of nudging is just not going to quite you know it's just going going to quite you know it's just going going to quite you know it's just going to keep dancing around the thing but to keep dancing around the thing but to keep dancing around the thing but it's not going to quite get there. it's not going to quite get there. it's not going to quite get there. >> Yeah. >> Yeah. >> Yeah. >> And um >> And um >> And um >> I found that like if you think about the >> I found that like if you think about the >> I found that like if you think about the what what I was just blown away with is what what I was just blown away with is what what I was just blown away with is the adherence to like the overall the adherence to like the overall the adherence to like the overall instructions and structure. M instructions and structure. M instructions and structure. M >> like I could not like cuz in a lot of >> like I could not like cuz in a lot of >> like I could not like cuz in a lot of ways that is really the hallmark of the ways that is really the hallmark of the ways that is really the hallmark of the best candidates at a company like Meta. best candidates at a company like Meta. best candidates at a company like Meta. It's like their ability to do the It's like their ability to do the It's like their ability to do the expectation setting up front to like set expectation setting up front to like set expectation setting up front to like set the right expectations to lay out their the right expectations to lay out their the right expectations to lay out their game plan for the interview and then to game plan for the interview and then to game plan for the interview and then to like actually hold themselves like actually hold themselves like actually hold themselves accountable and to stay on on on the accountable and to stay on on on the accountable and to stay on on on the path as they go through the interview path as they go through the interview path as they go through the interview and to way point along the way and check and to way point along the way and check and to way point along the way and check in with you. in with you. in with you. >> Like that was the thing that I kind of >> Like that was the thing that I kind of >> Like that was the thing that I kind of was most blown away with initially. It was most blown away with initially. It was most blown away with initially. It was less like the the the quality of the was less like the the the quality of the was less like the the the quality of the content that that it was generating, but content that that it was generating, but content that that it was generating, but it was its ability to get through the it was its ability to get through the it was its ability to get through the whole exercise by playing the game that whole exercise by playing the game that whole exercise by playing the game that I asked it to play I asked it to play I asked it to play >> and internalizing my framework down to >> and internalizing my framework down to >> and internalizing my framework down to like how it evaluates problems by like how it evaluates problems by like how it evaluates problems by scoring them on, you know, severity and scoring them on, you know, severity and scoring them on, you know, severity and frequency or picking um a segment based frequency or picking um a segment based frequency or picking um a segment based on reach and impact um you know, like or on reach and impact um you know, like or on reach and impact um you know, like or I'm sorry, reach in underserved degree I'm sorry, reach in underserved degree I'm sorry, reach in underserved degree relative to the mission. Um, so it's relative to the mission. Um, so it's relative to the mission. Um, so it's like just ability to understand the game like just ability to understand the game like just ability to understand the game I wanted it to play was by far the most I wanted it to play was by far the most I wanted it to play was by far the most impressive thing that I kind of made me impressive thing that I kind of made me impressive thing that I kind of made me get excited less it felt like the get excited less it felt like the get excited less it felt like the content itself was more of like um content itself was more of like um content itself was more of like um implementation detail that could be implementation detail that could be implementation detail that could be figured out later like getting the figured out later like getting the figured out later like getting the content to be better. content to be better. content to be better. >> So one of these light bulb moments was >> So one of these light bulb moments was >> So one of these light bulb moments was that the instructions were like the best that the instructions were like the best that the instructions were like the best case scenario of following the case scenario of following the case scenario of following the instructions. Now we need to tweak each instructions. Now we need to tweak each instructions. Now we need to tweak each and every step maybe to generate more and every step maybe to generate more and every step maybe to generate more like nuanced answers. But you saw it like nuanced answers. But you saw it like nuanced answers. But you saw it like going through to the letter of what like going through to the letter of what like going through to the letter of what you said is important as a meta you said is important as a meta you said is important as a meta interviewer. interviewer. interviewer. >> Yeah. Like and and I can't un this is >> Yeah. Like and and I can't un this is >> Yeah. Like and and I can't un this is one of those things that like people one of those things that like people one of those things that like people just get wrong in general even with just get wrong in general even with just get wrong in general even with human coaching is like when I tell them human coaching is like when I tell them human coaching is like when I tell them how to get ready for interviews. how to get ready for interviews. how to get ready for interviews. Some people just um they completely Some people just um they completely Some people just um they completely discount the value of having structure discount the value of having structure discount the value of having structure >> in interviews. And so the fact that I >> in interviews. And so the fact that I >> in interviews. And so the fact that I could get the the the the co-pilot, the could get the the the the co-pilot, the could get the the the the co-pilot, the the AI to do it like pretty early on was the AI to do it like pretty early on was the AI to do it like pretty early on was to me like a very strong signal because to me like a very strong signal because to me like a very strong signal because I just needed people to remember it's I just needed people to remember it's I just needed people to remember it's like there's no such thing as too much like there's no such thing as too much like there's no such thing as too much repetition when you're coaching someone. repetition when you're coaching someone. repetition when you're coaching someone. >> I just needed them like to know every >> I just needed them like to know every >> I just needed them like to know every question you work on, I do not want you question you work on, I do not want you question you work on, I do not want you to skip this step. Always start with to skip this step. Always start with to skip this step. Always start with assumptions. Always walk through your assumptions. Always walk through your assumptions. Always walk through your game plan and then get into the meat of game plan and then get into the meat of game plan and then get into the meat of it. it. it. >> Amazing. Okay. So now you have something >> Amazing. Okay. So now you have something >> Amazing. Okay. So now you have something sort of working if I understand sort of working if I understand sort of working if I understand correctly. you did like hundreds of correctly. you did like hundreds of correctly. you did like hundreds of iterations around that but you managed iterations around that but you managed iterations around that but you managed to get it through an interview process to get it through an interview process to get it through an interview process end to end. end to end. end to end. >> Um well the first very first version I

>> Um well the first very first version I >> Um well the first very first version I actually had one kind of project for actually had one kind of project for actually had one kind of project for both product sense and analytical both product sense and analytical both product sense and analytical thinking I thought I could I thought thinking I thought I could I thought thinking I thought I could I thought that was like why not that was like why not that was like why not >> um and then I I I pretty early on as >> um and then I I I pretty early on as >> um and then I I I pretty early on as part of that initial tinkering phase at part of that initial tinkering phase at part of that initial tinkering phase at some point some point some point >> by I I I branched them out into two >> by I I I branched them out into two >> by I I I branched them out into two separate projects. So, one project for separate projects. So, one project for separate projects. So, one project for product sense and one project for product sense and one project for product sense and one project for analytical thinking because I realized analytical thinking because I realized analytical thinking because I realized while the game is similar, while the game is similar, while the game is similar, >> the fact that there's different >> the fact that there's different >> the fact that there's different evaluation rubrics at the end that it evaluation rubrics at the end that it evaluation rubrics at the end that it should be holding itself accountable to should be holding itself accountable to should be holding itself accountable to for each of them made me realize that for each of them made me realize that for each of them made me realize that having two different kind of potentially having two different kind of potentially having two different kind of potentially um conflicting evaluation rubrics would um conflicting evaluation rubrics would um conflicting evaluation rubrics would would kind of pollute the the quality of would kind of pollute the the quality of would kind of pollute the the quality of the experience. So yeah, I would say the experience. So yeah, I would say the experience. So yeah, I would say like to answer your question, once I got like to answer your question, once I got like to answer your question, once I got out of that initial tinkering phase, I out of that initial tinkering phase, I out of that initial tinkering phase, I had two projects, had two projects, had two projects, >> one for product sense, one for >> one for product sense, one for >> one for product sense, one for analytical thinking and then the the analytical thinking and then the the analytical thinking and then the the once I kind of was like once I kind of was like once I kind of was like I think this could potentially pass an I think this could potentially pass an I think this could potentially pass an interview like I feel like cuz by the interview like I feel like cuz by the interview like I feel like cuz by the way the the big decision we haven't made way the the big decision we haven't made way the the big decision we haven't made yet is like I had the AI play the role yet is like I had the AI play the role yet is like I had the AI play the role of the co-pilot, not the I'm sorry, the of the co-pilot, not the I'm sorry, the of the co-pilot, not the I'm sorry, the AI play the role of the candidate, not AI play the role of the candidate, not AI play the role of the candidate, not the interviewer. And the reason for that the interviewer. And the reason for that the interviewer. And the reason for that is because I found myself in the shoes is because I found myself in the shoes is because I found myself in the shoes of a coach and in the shoes of an of a coach and in the shoes of an of a coach and in the shoes of an interviewer doing mock interviews with interviewer doing mock interviews with interviewer doing mock interviews with people all the time. people all the time. people all the time. >> So I wanted like the thing I was most >> So I wanted like the thing I was most >> So I wanted like the thing I was most kind of calibrated to like insensitized kind of calibrated to like insensitized kind of calibrated to like insensitized to is what is the performance of a to is what is the performance of a to is what is the performance of a candidate look like? candidate look like? candidate look like? >> And so I wanted the AI to play the role >> And so I wanted the AI to play the role >> And so I wanted the AI to play the role of a candidate by really and then I of a candidate by really and then I of a candidate by really and then I wanted people to be able to use the AI wanted people to be able to use the AI wanted people to be able to use the AI to to to >> to see what a good candidate behaves >> to see what a good candidate behaves >> to see what a good candidate behaves like. Um, so once I had that all up and like. Um, so once I had that all up and like. Um, so once I had that all up and running, I pinged a few um, people that running, I pinged a few um, people that running, I pinged a few um, people that I knew who were calibrated interviewers I knew who were calibrated interviewers I knew who were calibrated interviewers from Meta. from Meta. from Meta. >> I'm like, "Hey, I built this thing for >> I'm like, "Hey, I built this thing for >> I'm like, "Hey, I built this thing for my course. I'm about to give it to my my course. I'm about to give it to my my course. I'm about to give it to my students. I don't want to take them off students. I don't want to take them off students. I don't want to take them off track. Um, and lead them astray. Could track. Um, and lead them astray. Could track. Um, and lead them astray. Could you just like ask it a question? Ask it you just like ask it a question? Ask it you just like ask it a question? Ask it this question, right? Or or pick any this question, right? Or or pick any this question, right? Or or pick any product sense question that you like to product sense question that you like to product sense question that you like to ask in interviews. Um, and just like be ask in interviews. Um, and just like be ask in interviews. Um, and just like be play the role of the interviewer and play the role of the interviewer and play the role of the interviewer and just if you agree with what it's saying, just if you agree with what it's saying, just if you agree with what it's saying, say yes every time it checks in with say yes every time it checks in with say yes every time it checks in with you. And if you think it's not making you. And if you think it's not making you. And if you think it's not making sense, ask it a question like you would sense, ask it a question like you would sense, ask it a question like you would ask any other candidate. Can you help me ask any other candidate. Can you help me ask any other candidate. Can you help me understand why you picked this segment understand why you picked this segment understand why you picked this segment or I'm not really following your or I'm not really following your or I'm not really following your thinking on that mission statement or thinking on that mission statement or thinking on that mission statement or can you better you could you help me can you better you could you help me can you better you could you help me understand how this connects to the understand how this connects to the understand how this connects to the company's goals? Like you know, like company's goals? Like you know, like company's goals? Like you know, like whatever questions you'd ask in a real whatever questions you'd ask in a real whatever questions you'd ask in a real interview, ask it. Um, and then just interview, ask it. Um, and then just interview, ask it. Um, and then just like get let me know when you're done like get let me know when you're done like get let me know when you're done and and let me know kind of like your and and let me know kind of like your and and let me know kind of like your general assessment of, you know, like general assessment of, you know, like general assessment of, you know, like how was that? how was that? how was that? >> Um, and one of the first responses I got

>> Um, and one of the first responses I got >> Um, and one of the first responses I got I got back was uh in and in in slightly I got back was uh in and in in slightly I got back was uh in and in in slightly different word. I'm not going to say who different word. I'm not going to say who different word. I'm not going to say who it was, but it was uh someone who was it was, but it was uh someone who was it was, but it was uh someone who was definitely someone I worked with at definitely someone I worked with at definitely someone I worked with at Meta, who was a calibrated interviewer, Meta, who was a calibrated interviewer, Meta, who was a calibrated interviewer, is not at Meta anymore. Um, he was like, is not at Meta anymore. Um, he was like, is not at Meta anymore. Um, he was like, "Hey, like can I hire this co-pilot? "Hey, like can I hire this co-pilot? "Hey, like can I hire this co-pilot? Like I feel like this this would be like Like I feel like this this would be like Like I feel like this this would be like a passing score for me." a passing score for me." a passing score for me." >> Wow. Okay. Yeah, >> Wow. Okay. Yeah, >> Wow. Okay. Yeah, >> that that's incredible actually. I think >> that that's incredible actually. I think >> that that's incredible actually. I think like um like um like um >> it's and I think it's down to both you >> it's and I think it's down to both you >> it's and I think it's down to both you understanding what great looks like and understanding what great looks like and understanding what great looks like and also being able to actually calibrate it also being able to actually calibrate it also being able to actually calibrate it step by step to actually make it look step by step to actually make it look step by step to actually make it look great. And I think the separation you great. And I think the separation you great. And I think the separation you said I think it's very important to say said I think it's very important to say said I think it's very important to say like once you you branched out into two like once you you branched out into two like once you you branched out into two different things is exactly what I think different things is exactly what I think different things is exactly what I think when I'm saying okay if it's a when I'm saying okay if it's a when I'm saying okay if it's a completely different instruction process completely different instruction process completely different instruction process or step-by-step guide then this could be or step-by-step guide then this could be or step-by-step guide then this could be a good signal to think about training a good signal to think about training a good signal to think about training them differently. It's like trying to them differently. It's like trying to them differently. It's like trying to have a sales guy also do marketing. have a sales guy also do marketing. have a sales guy also do marketing. Okay. It's like two different use cases Okay. It's like two different use cases Okay. It's like two different use cases essentially. essentially. essentially. >> Well, that one u not to get pulled into >> Well, that one u not to get pulled into >> Well, that one u not to get pulled into too big of a tangent, but you know, some too big of a tangent, but you know, some too big of a tangent, but you know, some companies obviously see sales and companies obviously see sales and companies obviously see sales and marketing as needing to marketing as needing to marketing as needing to >> be woven out of the same fabric and play >> be woven out of the same fabric and play >> be woven out of the same fabric and play kind of like a consistent game, use the kind of like a consistent game, use the kind of like a consistent game, use the same voice and um and you know that's same voice and um and you know that's same voice and um and you know that's why they rolled them sometimes into like why they rolled them sometimes into like why they rolled them sometimes into like a chief revenue officer kind of role is a chief revenue officer kind of role is a chief revenue officer kind of role is because they don't want them to function because they don't want them to function because they don't want them to function as two completely different things. But as two completely different things. But as two completely different things. But um but but I think your point stands is um but but I think your point stands is um but but I think your point stands is like I think if there is a specific uh like I think if there is a specific uh like I think if there is a specific uh workflow or a specific context space workflow or a specific context space workflow or a specific context space that someone finds themselves in and you that someone finds themselves in and you that someone finds themselves in and you don't want that context space to get don't want that context space to get don't want that context space to get polluted with a different context space polluted with a different context space polluted with a different context space than being really intentional with I than being really intentional with I than being really intentional with I mean this is kind of like the modern mean this is kind of like the modern mean this is kind of like the modern version of scoping in a way like it's version of scoping in a way like it's version of scoping in a way like it's you're kind of scoping the environments you're kind of scoping the environments you're kind of scoping the environments in which things can happen in which things can happen in which things can happen non-deterministically. non-deterministically. non-deterministically. >> Yeah. But like you're creating the >> Yeah. But like you're creating the >> Yeah. But like you're creating the boundaries and when you're setting the boundaries and when you're setting the boundaries and when you're setting the boundaries quite tightly, I think I boundaries quite tightly, I think I boundaries quite tightly, I think I think that's a good signal to actually think that's a good signal to actually think that's a good signal to actually train on the on the process. Even though train on the on the process. Even though train on the on the process. Even though you can move within the boundaries left you can move within the boundaries left you can move within the boundaries left and right and you can do segmentation and right and you can do segmentation and right and you can do segmentation differently maybe, but it's still around differently maybe, but it's still around differently maybe, but it's still around the same framework. And I think that the same framework. And I think that the same framework. And I think that that's key. that's key. that's key. >> Yeah. But but and and importantly like >> Yeah. But but and and importantly like >> Yeah. But but and and importantly like the way that you do segmentation in a the way that you do segmentation in a the way that you do segmentation in a product sense interview is kind of at a product sense interview is kind of at a product sense interview is kind of at a different level of altitude and fidelity different level of altitude and fidelity different level of altitude and fidelity than the the way you do segmentation in than the the way you do segmentation in than the the way you do segmentation in analytical thinking interview. analytical thinking interview. analytical thinking interview. um like analytical thinking I want to I um like analytical thinking I want to I um like analytical thinking I want to I want you to stay at the level of high want you to stay at the level of high want you to stay at the level of high high level ecosystem players right it high level ecosystem players right it high level ecosystem players right it could it could even be as broad as like could it could even be as broad as like could it could even be as broad as like the supply and the demand side of the the supply and the demand side of the the supply and the demand side of the marketplace marketplace marketplace >> um but in a product sense interview >> um but in a product sense interview >> um but in a product sense interview that's kind of like step you know step that's kind of like step you know step that's kind of like step you know step one one one >> and then you want to kind of show me >> and then you want to kind of show me >> and then you want to kind of show me that you could identify that you could identify that you could identify >> several ecosystem players not just like >> several ecosystem players not just like >> several ecosystem players not just like the the key supply and demand side but I the the key supply and demand side but I the the key supply and demand side but I want to really understand your ecosystem want to really understand your ecosystem want to really understand your ecosystem thinking and then I want to see how um thinking and then I want to see how um thinking and then I want to see how um how you could break down how you could how you could break down how you could how you could break down how you could pick one of those ecosystem players and pick one of those ecosystem players and pick one of those ecosystem players and then use something other than just like then use something other than just like then use something other than just like boring demographic kind of breakdown boring demographic kind of breakdown boring demographic kind of breakdown segmentation heristuristics to craft segmentation heristuristics to craft segmentation heristuristics to craft unique segments unique segments unique segments >> that you could then put into some kind >> that you could then put into some kind >> that you could then put into some kind of persona and then brainstorm some of persona and then brainstorm some of persona and then brainstorm some problems for that persona along a problems for that persona along a problems for that persona along a journey and then land on some solutions journey and then land on some solutions journey and then land on some solutions and pick one and tell me about that and pick one and tell me about that and pick one and tell me about that right and so so like there and what I right and so so like there and what I right and so so like there and what I tell people in my course all the time is tell people in my course all the time is tell people in my course all the time is like analytical thinking and product like analytical thinking and product like analytical thinking and product sense it's almost like you're parking at sense it's almost like you're parking at sense it's almost like you're parking at the trail head. You're going to go on a the trail head. You're going to go on a the trail head. You're going to go on a on a hike in nature. Like you're hanging on a hike in nature. Like you're hanging on a hike in nature. Like you're hanging out. You're starting at the same point. out. You're starting at the same point. out. You're starting at the same point. You're going to start with some You're going to start with some You're going to start with some assumptions and set some rationale or assumptions and set some rationale or assumptions and set some rationale or motivation for why this matters, but motivation for why this matters, but motivation for why this matters, but then they're going to go in different then they're going to go in different then they're going to go in different directions the moment you start to kind directions the moment you start to kind directions the moment you start to kind of get into the step after that. of get into the step after that. of get into the step after that. >> Yeah. And I think Okay, so I'm going to >> Yeah. And I think Okay, so I'm going to >> Yeah. And I think Okay, so I'm going to back to the story. So you gave it to a back to the story. So you gave it to a back to the story. So you gave it to a few ex meta interviewers or meta few ex meta interviewers or meta few ex meta interviewers or meta interviewers. You got calibrated interviewers. You got calibrated interviewers. You got calibrated feedback saying, "Can I hire this feedback saying, "Can I hire this feedback saying, "Can I hire this person?" Which is amazing signal. Then person?" Which is amazing signal. Then person?" Which is amazing signal. Then what did you do? what did you do? what did you do? Yeah, it's also, I guess, like a Yeah, it's also, I guess, like a Yeah, it's also, I guess, like a testament to the the shortcomings of testament to the the shortcomings of testament to the the shortcomings of modern hiring practices, which is modern hiring practices, which is modern hiring practices, which is another thing we could talk about. Um, another thing we could talk about. Um, another thing we could talk about. Um, but you know, hey, if they want to play but you know, hey, if they want to play but you know, hey, if they want to play a game, like candidates are just need to a game, like candidates are just need to a game, like candidates are just need to learn to play the if the company is learn to play the if the company is learn to play the if the company is going to have candidates play a game, going to have candidates play a game, going to have candidates play a game, then you can't blame the candidates for then you can't blame the candidates for then you can't blame the candidates for trying to get really good at playing trying to get really good at playing trying to get really good at playing that game. that game. that game. >> Um, so what happened after that is I uh >> Um, so what happened after that is I uh >> Um, so what happened after that is I uh gave it so I had like I mentioned to gave it so I had like I mentioned to gave it so I had like I mentioned to you, I had already run two cohorts. you, I had already run two cohorts. you, I had already run two cohorts. >> Mh. >> Mh. >> Mh. >> Okay. So what I did is I before I >> Okay. So what I did is I before I >> Okay. So what I did is I before I started my January cohort, because this started my January cohort, because this started my January cohort, because this was all happening again over the course was all happening again over the course was all happening again over the course of like a two-eek period. So before I of like a two-eek period. So before I of like a two-eek period. So before I put it in the hands of my January cohort put it in the hands of my January cohort put it in the hands of my January cohort students, which was like a couple weeks students, which was like a couple weeks students, which was like a couple weeks out, I emailed all of my my quoteunquote out, I emailed all of my my quoteunquote out, I emailed all of my my quoteunquote alumni who took my my July and my alumni who took my my July and my alumni who took my my July and my October cohorts, and I said, "Hey, like October cohorts, and I said, "Hey, like October cohorts, and I said, "Hey, like this is free for you. I'm just giving this is free for you. I'm just giving this is free for you. I'm just giving this to you. It's like I I took all your this to you. It's like I I took all your this to you. It's like I I took all your feedback. I listened to all the feedback. I listened to all the feedback. I listened to all the questions you had. I'm thinking ahead to questions you had. I'm thinking ahead to questions you had. I'm thinking ahead to my next cohort. my next cohort. my next cohort. Some of you are still in interview prep. Some of you are still in interview prep. Some of you are still in interview prep. Some of you already landed offers. Some of you already landed offers. Some of you already landed offers. That's cool. But like regardless, it That's cool. But like regardless, it That's cool. But like regardless, it would be it would mean a lot to me if would be it would mean a lot to me if would be it would mean a lot to me if you could just like play with this and you could just like play with this and you could just like play with this and tell me how helpful is this? What what tell me how helpful is this? What what tell me how helpful is this? What what could be better? Um, so that's kind of could be better? Um, so that's kind of could be better? Um, so that's kind of like wave two is like putting in the like wave two is like putting in the like wave two is like putting in the hands of the alumni uh from the course hands of the alumni uh from the course hands of the alumni uh from the course and getting their feedback. and getting their feedback. and getting their feedback. So I think here if I highlight two So I think here if I highlight two So I think here if I highlight two things that are important is first of things that are important is first of things that are important is first of all you gave it to the interviewers all you gave it to the interviewers all you gave it to the interviewers essentially to make sure that the bar is essentially to make sure that the bar is essentially to make sure that the bar is high enough and then you gave it to the high enough and then you gave it to the high enough and then you gave it to the actual let's call it the persona who is actual let's call it the persona who is actual let's call it the persona who is going to actually use it to actually use going to actually use it to actually use going to actually use it to actually use it and see how this goes and to get that it and see how this goes and to get that it and see how this goes and to get that essential feel for whether this gives essential feel for whether this gives essential feel for whether this gives personalization at scale. Right. personalization at scale. Right. personalization at scale. Right. >> Okay. Totally. >> Okay. Totally. >> Okay. Totally. >> And then and then what did your alumni >> And then and then what did your alumni >> And then and then what did your alumni say? say? say? Um, I mean, like it was it was very positive, but like it was it was very positive, but I'm I'm trying to kind of frame this in I'm I'm trying to kind of frame this in I'm I'm trying to kind of frame this in a way that doesn't sound like I was just a way that doesn't sound like I was just a way that doesn't sound like I was just like I I genuinely was looking for like I I genuinely was looking for like I I genuinely was looking for what's wrong with it. what's wrong with it. what's wrong with it. >> And I did get like the overwhelming kind >> And I did get like the overwhelming kind >> And I did get like the overwhelming kind of feedback I got is like this is of feedback I got is like this is of feedback I got is like this is incredibly helpful. incredibly helpful. incredibly helpful. >> Wow. and >> Wow. and >> Wow. and the like I felt like the um you know the the like I felt like the um you know the the like I felt like the um you know the segmentation just felt a a little kind segmentation just felt a a little kind segmentation just felt a a little kind of like shallow to me or it didn't of like shallow to me or it didn't of like shallow to me or it didn't really follow like what you said or it really follow like what you said or it really follow like what you said or it picked the supply side of the ecosystem picked the supply side of the ecosystem picked the supply side of the ecosystem when I I think you generally recommend when I I think you generally recommend when I I think you generally recommend for this kind of question maybe to pick for this kind of question maybe to pick for this kind of question maybe to pick the demand side the demand side the demand side >> um cuz you know for like a consumer >> um cuz you know for like a consumer >> um cuz you know for like a consumer interview I generally recommend going interview I generally recommend going interview I generally recommend going deep in a product sense on like the deep in a product sense on like the deep in a product sense on like the demand side or the consumer side demand side or the consumer side demand side or the consumer side >> um and I was like okay yeah I never told >> um and I was like okay yeah I never told >> um and I was like okay yeah I never told the instructions that it should do that the instructions that it should do that the instructions that it should do that >> um or when it got into problem >> um or when it got into problem >> um or when it got into problem identification identification identification It didn't do like uh people were like, It didn't do like uh people were like, It didn't do like uh people were like, "Yeah, you always talk about how you "Yeah, you always talk about how you "Yeah, you always talk about how you should try to do a journey before you should try to do a journey before you should try to do a journey before you pick a problem." But the co-pilot wasn't pick a problem." But the co-pilot wasn't pick a problem." But the co-pilot wasn't doing a journey. It was just kind of doing a journey. It was just kind of doing a journey. It was just kind of going from like, you know, um a segment going from like, you know, um a segment going from like, you know, um a segment to like the problems. to like the problems. to like the problems. >> And so I I I actually want to get more >> And so I I I actually want to get more >> And so I I I actually want to get more practice with the journey piece, um etc. practice with the journey piece, um etc. practice with the journey piece, um etc. So I was I was starting to get kind of So I was I was starting to get kind of So I was I was starting to get kind of like specific pieces of feedback uh or like specific pieces of feedback uh or like specific pieces of feedback uh or like the mission statements were too like the mission statements were too like the mission statements were too verbose. M um so people were kind of verbose. M um so people were kind of verbose. M um so people were kind of like holding me accountable to like the like holding me accountable to like the like holding me accountable to like the stuff that I told them and they were stuff that I told them and they were stuff that I told them and they were like look like

like look like like look like >> I'm confused I'm confused because you >> I'm confused I'm confused because you >> I'm confused I'm confused because you just gave me this tool and like I'm just gave me this tool and like I'm just gave me this tool and like I'm trying to figure out whether what you trying to figure out whether what you trying to figure out whether what you told me is the source of truth or what told me is the source of truth or what told me is the source of truth or what the co-pilot is telling me is the source the co-pilot is telling me is the source the co-pilot is telling me is the source of truth. of truth. of truth. >> And that created an urgency for me to be >> And that created an urgency for me to be >> And that created an urgency for me to be like like I can't put this I can't like like I can't put this I can't like like I can't put this I can't like like like >> give this now to a bunch of new people >> give this now to a bunch of new people >> give this now to a bunch of new people in January if I'm going to contradict in January if I'm going to contradict in January if I'm going to contradict myself like if there's going to be myself like if there's going to be myself like if there's going to be contradictions in the system. Mhm. contradictions in the system. Mhm. contradictions in the system. Mhm. >> So that's when I kind of went into like >> So that's when I kind of went into like >> So that's when I kind of went into like intense tinkering mode for like phase intense tinkering mode for like phase intense tinkering mode for like phase two of tinkering, which is like, okay, two of tinkering, which is like, okay, two of tinkering, which is like, okay, now I'm going to literally go section by now I'm going to literally go section by now I'm going to literally go section by section in both of these. I'm going to section in both of these. I'm going to section in both of these. I'm going to look at the last five questions in the look at the last five questions in the look at the last five questions in the question bank from Louiswis Lynn. I'm question bank from Louiswis Lynn. I'm question bank from Louiswis Lynn. I'm going to drop them into new chats and going to drop them into new chats and going to drop them into new chats and I'm not going to basically like be done I'm not going to basically like be done I'm not going to basically like be done with this process until like until until with this process until like until until with this process until like until until there's no contradictions between what there's no contradictions between what there's no contradictions between what the co-pilot does and what I would do. the co-pilot does and what I would do. the co-pilot does and what I would do. And I'd say in the first phase I was a And I'd say in the first phase I was a And I'd say in the first phase I was a little bit more forgiving of it because little bit more forgiving of it because little bit more forgiving of it because I was like okay like some people might I was like okay like some people might I was like okay like some people might be okay be okay be okay >> with like >> with like >> with like >> it's just a co-pilot >> it's just a co-pilot >> it's just a co-pilot >> for me it's just like this this tool but >> for me it's just like this this tool but >> for me it's just like this this tool but like when people actually started being like when people actually started being like when people actually started being like I'm confused and I'm actually like like I'm confused and I'm actually like like I'm confused and I'm actually like I'm a little less confident in what I I'm a little less confident in what I I'm a little less confident in what I should do for this step because of this. should do for this step because of this. should do for this step because of this. >> I was like okay that's the stakes just >> I was like okay that's the stakes just >> I was like okay that's the stakes just went up and I need to like take up I I went up and I need to like take up I I went up and I need to like take up I I need to take the instructions way more need to take the instructions way more need to take the instructions way more seriously. seriously. seriously. >> Yeah. I think I think essentially at >> Yeah. I think I think essentially at >> Yeah. I think I think essentially at this point if you haven't done anything this point if you haven't done anything this point if you haven't done anything else it would detract the value of you else it would detract the value of you else it would detract the value of you as a coach essentially. It's like can I as a coach essentially. It's like can I as a coach essentially. It's like can I put my name on it? It's like first of put my name on it? It's like first of put my name on it? It's like first of all, okay, I can put it in front of all, okay, I can put it in front of all, okay, I can put it in front of interviewers. Okay, got a good thing, interviewers. Okay, got a good thing, interviewers. Okay, got a good thing, good signal, but can I really put my good signal, but can I really put my good signal, but can I really put my name on it? And I think that's that's a name on it? And I think that's that's a name on it? And I think that's that's a key moment in time where you say, okay, key moment in time where you say, okay, key moment in time where you say, okay, now it become it's ready to be now it become it's ready to be now it become it's ready to be productized essentially where I can put productized essentially where I can put productized essentially where I can put my name on it, right? And that that was my name on it, right? And that that was my name on it, right? And that that was think that that was the point you had. think that that was the point you had. think that that was the point you had. >> Yeah, for sure. And um and look, I I >> Yeah, for sure. And um and look, I I >> Yeah, for sure. And um and look, I I take I take like being helpful to people take I take like being helpful to people take I take like being helpful to people like I can't I can't I can't overstate like I can't I can't I can't overstate like I can't I can't I can't overstate how how important it is to me that um how how important it is to me that um how how important it is to me that um people walk away more confident and more people walk away more confident and more people walk away more confident and more clear on what they need to do after clear on what they need to do after clear on what they need to do after working with me or taking my course. And working with me or taking my course. And working with me or taking my course. And so, you know, um if I had like a neutral so, you know, um if I had like a neutral so, you know, um if I had like a neutral impact on people, they're like, "Yeah, impact on people, they're like, "Yeah, impact on people, they're like, "Yeah, it's like, you know, whatever." Like I it's like, you know, whatever." Like I it's like, you know, whatever." Like I was kind of I would take that one way. was kind of I would take that one way. was kind of I would take that one way. And if they were just like, "It's And if they were just like, "It's And if they were just like, "It's perfect," that's one thing. But if but perfect," that's one thing. But if but perfect," that's one thing. But if but whenever someone tells me that they're whenever someone tells me that they're whenever someone tells me that they're um that they're more confused um that they're more confused um that they're more confused because of something I gave them, that's because of something I gave them, that's because of something I gave them, that's where I'm kind of like I can't put my where I'm kind of like I can't put my where I'm kind of like I can't put my name on that. Like that's that's like a name on that. Like that's that's like a name on that. Like that's that's like a quality bar that I hold for myself is quality bar that I hold for myself is quality bar that I hold for myself is like I I refuse to like put things out like I I refuse to like put things out like I I refuse to like put things out there that will hurt people's there that will hurt people's there that will hurt people's preparation in any way, which sounds preparation in any way, which sounds preparation in any way, which sounds obvious, but like obvious, but like obvious, but like >> I let's just say I don't think that's >> I let's just say I don't think that's >> I let's just say I don't think that's like the I I think I might have a like the I I think I might have a like the I I think I might have a different bar for that than some other different bar for that than some other different bar for that than some other people in the space. people in the space. people in the space. >> Mhm. But you didn't stop there, right? >> Mhm. But you didn't stop there, right? >> Mhm. But you didn't stop there, right? So even after you tweaked it a lot and So even after you tweaked it a lot and So even after you tweaked it a lot and did a lot of a lot of feedback after did a lot of a lot of feedback after did a lot of a lot of feedback after your students feedback, you continue to your students feedback, you continue to your students feedback, you continue to improve it, right? You had the ultimate improve it, right? You had the ultimate improve it, right? You had the ultimate test of making sure the co-pilot works. test of making sure the co-pilot works. test of making sure the co-pilot works. >> Yeah. And and I'd say right around the

>> Yeah. And and I'd say right around the >> Yeah. And and I'd say right around the time where I was like comfortable giving time where I was like comfortable giving time where I was like comfortable giving it to my January students. That's when I it to my January students. That's when I it to my January students. That's when I felt comfortable increasing the price of felt comfortable increasing the price of felt comfortable increasing the price of the course the course the course >> to reflect to like actually position put >> to reflect to like actually position put >> to reflect to like actually position put co-pilot in the name. co-pilot in the name. co-pilot in the name. >> The reason I called it co-pilot is >> The reason I called it co-pilot is >> The reason I called it co-pilot is because Tal Raviv was marketing his because Tal Raviv was marketing his because Tal Raviv was marketing his copilot stuff at the time. copilot stuff at the time. copilot stuff at the time. I talked to Tal I talked to Tal about it I talked to Tal I talked to Tal about it I talked to Tal I talked to Tal about it and Tal was like just just like put and Tal was like just just like put and Tal was like just just like put C-Pilot in the title. C-Pilot in the title. C-Pilot in the title. >> Um >> Um >> Um >> and um and then so that allowed me to go >> and um and then so that allowed me to go >> and um and then so that allowed me to go from like $600 to $1,000 for the course from like $600 to $1,000 for the course from like $600 to $1,000 for the course price. price. price. >> And I said, "Okay, like >> And I said, "Okay, like >> And I said, "Okay, like um um um what if someone doesn't want to take my what if someone doesn't want to take my what if someone doesn't want to take my course, but they want to get access to course, but they want to get access to course, but they want to get access to this tool, do I want to let people get this tool, do I want to let people get this tool, do I want to let people get access to it?" M access to it?" M access to it?" M >> um and I was like, "Okay, I I'll sell it >> um and I was like, "Okay, I I'll sell it >> um and I was like, "Okay, I I'll sell it separately and then I'll just bundle it separately and then I'll just bundle it separately and then I'll just bundle it in the course." in the course." in the course." >> But for me to sell it separately, to >> But for me to sell it separately, to >> But for me to sell it separately, to your point, I did have to be that was your point, I did have to be that was your point, I did have to be that was like my first time ever selling a like my first time ever selling a like my first time ever selling a digital product like this. So, um I digital product like this. So, um I digital product like this. So, um I didn't expect anyone to buy it. I didn't expect anyone to buy it. I didn't expect anyone to buy it. I thought I I just But then it started to thought I I just But then it started to thought I I just But then it started to happen because I linked to it from the happen because I linked to it from the happen because I linked to it from the the the Maven course page the the Maven course page the the Maven course page >> because I was like, "Hey, like this >> because I was like, "Hey, like this >> because I was like, "Hey, like this thing is included in the price, right?" thing is included in the price, right?" thing is included in the price, right?" So people would go and check it out. So people would go and check it out. So people would go and check it out. Maybe they didn't want the course, but Maybe they didn't want the course, but Maybe they didn't want the course, but they bought the co-pilot. Um, and I told they bought the co-pilot. Um, and I told they bought the co-pilot. Um, and I told people if they buy the co-pilot, then people if they buy the co-pilot, then people if they buy the co-pilot, then I'll just give them a discount in the I'll just give them a discount in the I'll just give them a discount in the amount of the co-pilot, which the the amount of the co-pilot, which the the amount of the co-pilot, which the the original price was $2.99. So I basically original price was $2.99. So I basically original price was $2.99. So I basically like if you buy it, I'll give you a $300 like if you buy it, I'll give you a $300 like if you buy it, I'll give you a $300 promo code on the course. So you'll promo code on the course. So you'll promo code on the course. So you'll you'll be made whole if you end up you'll be made whole if you end up you'll be made whole if you end up wanting the whole thing after. wanting the whole thing after. wanting the whole thing after. >> Yeah. So you you don't have anything to >> Yeah. So you you don't have anything to >> Yeah. So you you don't have anything to lose if you decide to go in this route. lose if you decide to go in this route. lose if you decide to go in this route. You could just try it out. And I think You could just try it out. And I think You could just try it out. And I think uh I think that's that's a great signal uh I think that's that's a great signal uh I think that's that's a great signal where at the end of the day, someone was where at the end of the day, someone was where at the end of the day, someone was paying for that advice. And did you get paying for that advice. And did you get paying for that advice. And did you get any refunds at that point? any refunds at that point? any refunds at that point? I mean, I think I've only ever had I mean, I think I've only ever had I mean, I think I've only ever had two two two >> co-pilot refunds over hundreds of sales >> co-pilot refunds over hundreds of sales >> co-pilot refunds over hundreds of sales at this point. at this point. at this point. >> Um, the course has refunds, but every >> Um, the course has refunds, but every >> Um, the course has refunds, but every instructor will instructor will instructor will >> tell you about that. I think there's >> tell you about that. I think there's >> tell you about that. I think there's like a lot of students that will just like a lot of students that will just like a lot of students that will just sign up for a course not reading the sign up for a course not reading the sign up for a course not reading the page, not knowing what it is. They see a page, not knowing what it is. They see a page, not knowing what it is. They see a guarantee from Maven, and they'll just guarantee from Maven, and they'll just guarantee from Maven, and they'll just sign up, not thinking twice about it, sign up, not thinking twice about it, sign up, not thinking twice about it, and then, and then, and then, >> you know, they'll just tell themselves a >> you know, they'll just tell themselves a >> you know, they'll just tell themselves a story about why that was a bad purchase. story about why that was a bad purchase. story about why that was a bad purchase. they'll get cold feet and they'll want they'll get cold feet and they'll want they'll get cold feet and they'll want their money back. But it's it's never their money back. But it's it's never their money back. But it's it's never because like I'm not delivering what I because like I'm not delivering what I because like I'm not delivering what I said I was delivering. It was more said I was delivering. It was more said I was delivering. It was more because they had some story in their because they had some story in their because they had some story in their head about what it would be. head about what it would be. head about what it would be. >> And instructors generally see somewhere >> And instructors generally see somewhere >> And instructors generally see somewhere between like 5 to 10% refund rates on between like 5 to 10% refund rates on between like 5 to 10% refund rates on their their Maven courses. their their Maven courses. their their Maven courses. >> Okay. So when when was the moment that >> Okay. So when when was the moment that >> Okay. So when when was the moment that like this co-pilot exploded? >> It was probably April when I had my >> It was probably April when I had my April 2025 when I had my um my Lenny April 2025 when I had my um my Lenny April 2025 when I had my um my Lenny guest post go live about Product Sense guest post go live about Product Sense guest post go live about Product Sense interviews. interviews. interviews. because that linked to all my things because that linked to all my things because that linked to all my things that linked to my course that linked to that linked to my course that linked to that linked to my course that linked to my templates that linked to my co-pilot my templates that linked to my co-pilot my templates that linked to my co-pilot um and uh that also basically um and uh that also basically um and uh that also basically established my number one spot on like established my number one spot on like established my number one spot on like you know SEO if someone looks for you know SEO if someone looks for you know SEO if someone looks for product sense interview and my my my product sense interview and my my my product sense interview and my my my stuff just pops up at the top. stuff just pops up at the top. stuff just pops up at the top. >> Um >> Um >> Um >> yeah, I think I did like $10,000 of >> yeah, I think I did like $10,000 of >> yeah, I think I did like $10,000 of co-pilot sales that that April. Did did co-pilot sales that that April. Did did co-pilot sales that that April. Did did you did you use the co-pilot when you you did you use the co-pilot when you you did you use the co-pilot when you wrote the Lennier articles? wrote the Lennier articles? wrote the Lennier articles? >> Uh for the >> Uh for the >> Uh for the >> I did I would say I did it much better >> I did I would say I did it much better >> I did I would say I did it much better for the sec for the analytical thinking for the sec for the analytical thinking for the sec for the analytical thinking one. one. one. >> Um again I think the analytical thinking >> Um again I think the analytical thinking >> Um again I think the analytical thinking stuff I'm just generally more proud of stuff I'm just generally more proud of stuff I'm just generally more proud of what the co-pilot does with analytical what the co-pilot does with analytical what the co-pilot does with analytical thinking. Um, so I felt more thinking. Um, so I felt more thinking. Um, so I felt more comfortable, I don't say delegating, but comfortable, I don't say delegating, but comfortable, I don't say delegating, but really like really like really like >> we we we like went big on examples in >> we we we like went big on examples in >> we we we like went big on examples in the analytical thinking post. Basically the analytical thinking post. Basically the analytical thinking post. Basically did one example. did one example. did one example. >> It's a great >> It's a great >> It's a great >> but then linked to two separate examples >> but then linked to two separate examples >> but then linked to two separate examples um in the in in sheets and the content um in the in in sheets and the content um in the in in sheets and the content of the sheets that were populated for of the sheets that were populated for of the sheets that were populated for those were all generated with my those were all generated with my those were all generated with my co-pilot. co-pilot. co-pilot. >> Um, for the product sense one >> Um, for the product sense one >> Um, for the product sense one >> I did generate the content with my >> I did generate the content with my >> I did generate the content with my co-pilot but I did a lot more kind of co-pilot but I did a lot more kind of co-pilot but I did a lot more kind of like iteration and tweaking. uh to to like iteration and tweaking. uh to to like iteration and tweaking. uh to to kind of get it ready to go live. But the kind of get it ready to go live. But the kind of get it ready to go live. But the I think the thing I was telling you when I think the thing I was telling you when I think the thing I was telling you when we were kind of doing our uh preparation we were kind of doing our uh preparation we were kind of doing our uh preparation for the episode is um when for the episode is um when for the episode is um when when Lenny and I started working on when Lenny and I started working on when Lenny and I started working on those posts, those posts, those posts, I I I it wasn't like to try to save myself it wasn't like to try to save myself it wasn't like to try to save myself time that I wanted to leverage the time that I wanted to leverage the time that I wanted to leverage the co-pilots, but it's more like I wanted co-pilots, but it's more like I wanted co-pilots, but it's more like I wanted to use my co-pilot I wanted to use the to use my co-pilot I wanted to use the to use my co-pilot I wanted to use the Lenny post as a forcing function Lenny post as a forcing function Lenny post as a forcing function >> to make my co-pilots better because I >> to make my co-pilots better because I >> to make my co-pilots better because I expected a lot more people to come and expected a lot more people to come and expected a lot more people to come and buy them. So there was buy them. So there was buy them. So there was >> a lot more people to come to my course. >> a lot more people to come to my course. >> a lot more people to come to my course. >> So essentially there's like this is the >> So essentially there's like this is the >> So essentially there's like this is the golden standard. If I can make this here golden standard. If I can make this here golden standard. If I can make this here as a forcing function, I think you'd be as a forcing function, I think you'd be as a forcing function, I think you'd be comfortable selling it, people using it comfortable selling it, people using it comfortable selling it, people using it because you use that for what's because you use that for what's because you use that for what's considered the golden standard and if considered the golden standard and if considered the golden standard and if you are proud of it and not just putting you are proud of it and not just putting you are proud of it and not just putting your name on it, but really proud of it your name on it, but really proud of it your name on it, but really proud of it being out there and I think that's a being out there and I think that's a being out there and I think that's a great signal. So great signal. So great signal. So >> yeah, I think Lenny Lenny has arguably >> yeah, I think Lenny Lenny has arguably >> yeah, I think Lenny Lenny has arguably like the most premium editorial voice of like the most premium editorial voice of like the most premium editorial voice of anyone that that I'm aware of that does anyone that that I'm aware of that does anyone that that I'm aware of that does like in our space. So yeah, my my idea like in our space. So yeah, my my idea like in our space. So yeah, my my idea was if I'd be comfortable putting my was if I'd be comfortable putting my was if I'd be comfortable putting my name under this uh behind this and like name under this uh behind this and like name under this uh behind this and like it would go live on Lenny's, then it it would go live on Lenny's, then it it would go live on Lenny's, then it would be good enough for like basically would be good enough for like basically would be good enough for like basically everyone else to use it.

everyone else to use it. everyone else to use it. >> Amazing. So, we touched a lot of things >> Amazing. So, we touched a lot of things >> Amazing. So, we touched a lot of things about your case study, but I want to about your case study, but I want to about your case study, but I want to help our listeners maybe think about help our listeners maybe think about help our listeners maybe think about like Ben's playbook for building a like Ben's playbook for building a like Ben's playbook for building a co-pilot. And I want us to try and think co-pilot. And I want us to try and think co-pilot. And I want us to try and think about like the main principles that we about like the main principles that we about like the main principles that we talked about here and try to help our talked about here and try to help our talked about here and try to help our listeners think about their problems and listeners think about their problems and listeners think about their problems and whether a co-pilot is good or bad for whether a co-pilot is good or bad for whether a co-pilot is good or bad for this situation. So, if if I had to ask this situation. So, if if I had to ask this situation. So, if if I had to ask you, what would you say would be like you, what would you say would be like you, what would you say would be like the biggest principles you see? the biggest principles you see? the biggest principles you see? I would say the first one is to like I would say the first one is to like I would say the first one is to like really sensitize yourself to what you really sensitize yourself to what you really sensitize yourself to what you find what what kind of advice or what find what what kind of advice or what find what what kind of advice or what kind of um expertise you find yourself kind of um expertise you find yourself kind of um expertise you find yourself using the most using the most using the most >> where >> where >> where >> you would be comfortable saying I might >> you would be comfortable saying I might >> you would be comfortable saying I might be like the best person at explaining be like the best person at explaining be like the best person at explaining this thing. Okay. Or there's other great this thing. Okay. Or there's other great this thing. Okay. Or there's other great people at explaining this thing but for people at explaining this thing but for people at explaining this thing but for some reason the way I explain it or the some reason the way I explain it or the some reason the way I explain it or the way I you know tackle it seems to really way I you know tackle it seems to really way I you know tackle it seems to really resonate with people. So trying to like resonate with people. So trying to like resonate with people. So trying to like really sensitize yourself to where you really sensitize yourself to where you really sensitize yourself to where you might have a superpower of some kind. might have a superpower of some kind. might have a superpower of some kind. Okay. Okay. Okay. >> Um and the reason that's important is >> Um and the reason that's important is >> Um and the reason that's important is because everything we talked about because everything we talked about because everything we talked about relied on my ability to evaluate the relied on my ability to evaluate the relied on my ability to evaluate the quality of the output. quality of the output. quality of the output. >> Yeah. >> Yeah. >> Yeah. >> Okay. And if I was trying to do >> Okay. And if I was trying to do >> Okay. And if I was trying to do something that I was not an expert on or something that I was not an expert on or something that I was not an expert on or that I wasn't extremely opinionated with that I wasn't extremely opinionated with that I wasn't extremely opinionated with real proof points that my opinionated real proof points that my opinionated real proof points that my opinionated beliefs were backed by real results, um beliefs were backed by real results, um beliefs were backed by real results, um then I would be uh shooting in the dark then I would be uh shooting in the dark then I would be uh shooting in the dark or I would have no ground to stand on as or I would have no ground to stand on as or I would have no ground to stand on as like a credible voice. And if you're like a credible voice. And if you're like a credible voice. And if you're just trying to build something for like just trying to build something for like just trying to build something for like for fun or just like you know as a for fun or just like you know as a for fun or just like you know as a hobbyist or you're not trying to like hobbyist or you're not trying to like hobbyist or you're not trying to like take a like a high stakes thing like take a like a high stakes thing like take a like a high stakes thing like interview preparation which is super interview preparation which is super interview preparation which is super high stakes then you know maybe you high stakes then you know maybe you high stakes then you know maybe you could like loosen the the requirements could like loosen the the requirements could like loosen the the requirements around that a bit. But if you if you are around that a bit. But if you if you are around that a bit. But if you if you are trying to bring your expertise into some trying to bring your expertise into some trying to bring your expertise into some kind of packaging that um people would kind of packaging that um people would kind of packaging that um people would pay for and maybe even command a premium pay for and maybe even command a premium pay for and maybe even command a premium price for then I would really try to price for then I would really try to price for then I would really try to like narrow in on that thing that you're like narrow in on that thing that you're like narrow in on that thing that you're uniquely positioned to do. And then um I uniquely positioned to do. And then um I uniquely positioned to do. And then um I would like we said establish a baseline. would like we said establish a baseline. would like we said establish a baseline. >> Okay. The re why use clawed projects

>> Okay. The re why use clawed projects >> Okay. The re why use clawed projects versus build software or you know in versus build software or you know in versus build software or you know in 2026 right? 2026 right? 2026 right? >> Yeah. Um, >> Yeah. Um, >> Yeah. Um, I would still argue that, I would still argue that, I would still argue that, okay, I did spend about four or five okay, I did spend about four or five okay, I did spend about four or five months last year with a really good months last year with a really good months last year with a really good engineer friend of mine building kind of engineer friend of mine building kind of engineer friend of mine building kind of like our own, like our own, like our own, >> you know, GPT wrapper, um, or our own, >> you know, GPT wrapper, um, or our own, >> you know, GPT wrapper, um, or our own, >> think of it like, you know, um, a Claude >> think of it like, you know, um, a Claude >> think of it like, you know, um, a Claude project software product that was using project software product that was using project software product that was using API token keys. Um, and we just started API token keys. Um, and we just started API token keys. Um, and we just started playing with it to get a feel for like, playing with it to get a feel for like, playing with it to get a feel for like, hey, like what if we could fully own hey, like what if we could fully own hey, like what if we could fully own this experience and build like a this experience and build like a this experience and build like a co-pilot platform, right? I was like, co-pilot platform, right? I was like, co-pilot platform, right? I was like, maybe that's the direction this all goes maybe that's the direction this all goes maybe that's the direction this all goes as I build a co-pilot platform. Um, and as I build a co-pilot platform. Um, and as I build a co-pilot platform. Um, and let me just tell everyone right now, let me just tell everyone right now, let me just tell everyone right now, like like like the amount of money that you're saving the amount of money that you're saving the amount of money that you're saving by using Claude's products or Chat GPT's by using Claude's products or Chat GPT's by using Claude's products or Chat GPT's products or Gemini's products, like the products or Gemini's products, like the products or Gemini's products, like the the sheer subsidies for tokens that we the sheer subsidies for tokens that we the sheer subsidies for tokens that we get as consumers is unimaginable until get as consumers is unimaginable until get as consumers is unimaginable until you start playing with these things with you start playing with these things with you start playing with these things with your own product. your own product. your own product. >> Yeah. >> Yeah. >> Yeah. >> So, that direction died very fast for >> So, that direction died very fast for >> So, that direction died very fast for me. Um, so that's kind of why I I me. Um, so that's kind of why I I me. Um, so that's kind of why I I wouldn't recommend necessarily building wouldn't recommend necessarily building wouldn't recommend necessarily building a new product for for your thing. And I a new product for for your thing. And I a new product for for your thing. And I do see people build their own co-pilots do see people build their own co-pilots do see people build their own co-pilots and their own stuff and and their own stuff and and their own stuff and >> because it's cool. >> because it's cool. >> because it's cool. >> It's cool. It's fun, but like but it's >> It's cool. It's fun, but like but it's >> It's cool. It's fun, but like but it's completely the wrong ide. I think uh for us to try building it I think uh for us to try building it like like that. like like that. like like that. >> Yeah. And and like I'll take I'll take >> Yeah. And and like I'll take I'll take >> Yeah. And and like I'll take I'll take it even a step further. Like it even a step further. Like it even a step further. Like the better the product and the better the better the product and the better the better the product and the better the co-pilot, the more usage it's going the co-pilot, the more usage it's going the co-pilot, the more usage it's going to get, which means the more token heavy to get, which means the more token heavy to get, which means the more token heavy the consumption will be. the consumption will be. the consumption will be. >> Yeah. >> Yeah. >> Yeah. >> And if people come to if you're like, >> And if people come to if you're like, >> And if people come to if you're like, oh, like we're not we're going to build oh, like we're not we're going to build oh, like we're not we're going to build our own because like the chats won't our own because like the chats won't our own because like the chats won't even be that long or we're not going to even be that long or we're not going to even be that long or we're not going to even need that many tokens. It's like, even need that many tokens. It's like, even need that many tokens. It's like, well, does that mean like people don't well, does that mean like people don't well, does that mean like people don't actually value like are you basically actually value like are you basically actually value like are you basically building something so useless that building something so useless that building something so useless that people don't want to use it that much? people don't want to use it that much? people don't want to use it that much? Um, so the usage I was getting, I was Um, so the usage I was getting, I was Um, so the usage I was getting, I was regularly hitting usage limits in claude regularly hitting usage limits in claude regularly hitting usage limits in claude projects before they had their $200 projects before they had their $200 projects before they had their $200 tier. So I I would have to like stop in tier. So I I would have to like stop in tier. So I I would have to like stop in the middle of an interview practice, the middle of an interview practice, the middle of an interview practice, like an session, like an session, like an session, >> and I'd have to come back like four >> and I'd have to come back like four >> and I'd have to come back like four hours later and like continue my hours later and like continue my hours later and like continue my tinkering because I couldn't get it to tinkering because I couldn't get it to tinkering because I couldn't get it to generate more results. generate more results. generate more results. >> Um, so that that was the kind of like >> Um, so that that was the kind of like >> Um, so that that was the kind of like heaviness of usage that I was getting in heaviness of usage that I was getting in heaviness of usage that I was getting in my tinkering. And then my students were my tinkering. And then my students were my tinkering. And then my students were like, "Hey, like how can I get around like, "Hey, like how can I get around like, "Hey, like how can I get around these usage limits?" And I'm like, okay, these usage limits?" And I'm like, okay, these usage limits?" And I'm like, okay, if they're hitting the usage limits, if they're hitting the usage limits, if they're hitting the usage limits, that means that they're really playing that means that they're really playing that means that they're really playing with this. Yeah. Right. with this. Yeah. Right. with this. Yeah. Right. >> Um, so for all those reasons, I would >> Um, so for all those reasons, I would >> Um, so for all those reasons, I would say like if you're just getting started, say like if you're just getting started, say like if you're just getting started, use like the out of the box things that use like the out of the box things that use like the out of the box things that you can get from Claude or even like a you can get from Claude or even like a you can get from Claude or even like a custom GPT. Although I have I don't custom GPT. Although I have I don't custom GPT. Although I have I don't think the custom GPT exp I I do offer think the custom GPT exp I I do offer think the custom GPT exp I I do offer custom GPTs to people that buy the custom GPTs to people that buy the custom GPTs to people that buy the co-pilot, but I generally point people co-pilot, but I generally point people co-pilot, but I generally point people in the direction of claude projects or in the direction of claude projects or in the direction of claude projects or or gems. I think those are probably the or gems. I think those are probably the or gems. I think those are probably the two best best ways to do this. But just two best best ways to do this. But just two best best ways to do this. But just start giving it just like where we start giving it just like where we start giving it just like where we started like let's say you want to started like let's say you want to started like let's say you want to um let's pick on um let's pick on um let's pick on uh running a user research interview or uh running a user research interview or uh running a user research interview or something. something. something. >> Okay. Something I'm also like >> Okay. Something I'm also like >> Okay. Something I'm also like >> when I was thinking about what to teach >> when I was thinking about what to teach >> when I was thinking about what to teach like it was actually between teaching a like it was actually between teaching a like it was actually between teaching a course about user research and talking course about user research and talking course about user research and talking to customers and like interview prep. So to customers and like interview prep. So to customers and like interview prep. So like I I have a lot of opinions about like I I have a lot of opinions about like I I have a lot of opinions about user research. Okay. Um but let's say I user research. Okay. Um but let's say I user research. Okay. Um but let's say I wanted to build a co-pilot for user wanted to build a co-pilot for user wanted to build a co-pilot for user research and I wanted to productize research and I wanted to productize research and I wanted to productize that. I would start by, you know, asking that. I would start by, you know, asking that. I would start by, you know, asking um asking the co the the AI to play the um asking the co the the AI to play the um asking the co the the AI to play the role of a um I would tell it who who the role of a um I would tell it who who the role of a um I would tell it who who the AI is. I'd be like, you're this person. AI is. I'd be like, you're this person. AI is. I'd be like, you're this person. Here's your job. Here's where you live. Here's your job. Here's where you live. Here's your job. Here's where you live. >> Yeah. >> Yeah. >> Yeah. >> I I would I would make sure it knew knew >> I I would I would make sure it knew knew >> I I would I would make sure it knew knew who it was who it was who it was >> and then I would be like, I'm about to >> and then I would be like, I'm about to >> and then I would be like, I'm about to do a user research interview with you do a user research interview with you do a user research interview with you >> and I want to um I want you to answer >> and I want to um I want you to answer >> and I want to um I want you to answer these questions as if you're a these questions as if you're a these questions as if you're a >> participant. So I'd say like once you >> participant. So I'd say like once you >> participant. So I'd say like once you know what that thing is, I would I would know what that thing is, I would I would know what that thing is, I would I would try to break it down into like what's try to break it down into like what's try to break it down into like what's the the the >> what's the workflow in which that >> what's the workflow in which that >> what's the workflow in which that expertise fits. expertise fits. expertise fits. >> What's so for interview prep, it's >> What's so for interview prep, it's >> What's so for interview prep, it's generally like, you know, someone's generally like, you know, someone's generally like, you know, someone's getting ready for an interview. They may getting ready for an interview. They may getting ready for an interview. They may take my course, they may not take my take my course, they may not take my take my course, they may not take my course. They're trying to get ready. course. They're trying to get ready. course. They're trying to get ready. There's probably going to be mocks at There's probably going to be mocks at There's probably going to be mocks at some point. So where where in that some point. So where where in that some point. So where where in that process does this thing fit?

process does this thing fit? process does this thing fit? >> Um and where does your tool fit? Do you >> Um and where does your tool fit? Do you >> Um and where does your tool fit? Do you want them to use use it during an want them to use use it during an want them to use use it during an activity, before an activity, after an activity, before an activity, after an activity, before an activity, after an activity? So, trying to really be activity? So, trying to really be activity? So, trying to really be opinionated about how it might like when opinionated about how it might like when opinionated about how it might like when does it fit? does it fit? does it fit? >> Um, yeah, I would say that's probably >> Um, yeah, I would say that's probably >> Um, yeah, I would say that's probably step two. step two. step two. >> Okay. So, step number one is like >> Okay. So, step number one is like >> Okay. So, step number one is like understanding what good looks like and understanding what good looks like and understanding what good looks like and the fact that the results that you're the fact that the results that you're the fact that the results that you're bringing as a human being actually bringing as a human being actually bringing as a human being actually produces great results in the real world produces great results in the real world produces great results in the real world and that you know how to determine and that you know how to determine and that you know how to determine whether this is a bad answer or a good whether this is a bad answer or a good whether this is a bad answer or a good answer. That's step number one. Step answer. That's step number one. Step answer. That's step number one. Step number two is actually mapping the user number two is actually mapping the user number two is actually mapping the user journey of your users and understanding journey of your users and understanding journey of your users and understanding where exactly in this process does this where exactly in this process does this where exactly in this process does this fit like is there is this a specific fit like is there is this a specific fit like is there is this a specific step before during or after and like step before during or after and like step before during or after and like really nail down like the step that you really nail down like the step that you really nail down like the step that you need co-pilot or you could use copilot need co-pilot or you could use copilot need co-pilot or you could use copilot right right right >> you know uh I'm realizing as we're >> you know uh I'm realizing as we're >> you know uh I'm realizing as we're talking about this like we're basically talking about this like we're basically talking about this like we're basically like talking about my product sense like talking about my product sense like talking about my product sense framework right like that's you're you framework right like that's you're you framework right like that's you're you got to figure out what the thing is what got to figure out what the thing is what got to figure out what the thing is what you know who is it for what problem are you know who is it for what problem are you know who is it for what problem are you trying to solve for them in in the you trying to solve for them in in the you trying to solve for them in in the context of what they do and then what's context of what they do and then what's context of what they do and then what's the what's the angle in which you want the what's the angle in which you want the what's the angle in which you want to solve that problem. That's basically to solve that problem. That's basically to solve that problem. That's basically what we're talking about, right? That what we're talking about, right? That what we're talking about, right? That that is the playbook. that is the playbook. that is the playbook. >> Yeah. >> Yeah. >> Yeah. >> Um so yeah. >> Um so yeah. >> Um so yeah. >> Yeah. But I think another thing that you >> Yeah. But I think another thing that you >> Yeah. But I think another thing that you that that came up during the case study that that came up during the case study that that came up during the case study is the fact that you could build it is the fact that you could build it is the fact that you could build it bottom up, step by step. I think that's bottom up, step by step. I think that's bottom up, step by step. I think that's a really important thing is that you can a really important thing is that you can a really important thing is that you can target a specific step within the target a specific step within the target a specific step within the process and actually improve it. process and actually improve it. process and actually improve it. actually tweak it until you are actually tweak it until you are actually tweak it until you are satisfied with this specific step and satisfied with this specific step and satisfied with this specific step and then move on. Right. then move on. Right. then move on. Right. >> Yes. But let me clarify what um Okay. So >> Yes. But let me clarify what um Okay. So >> Yes. But let me clarify what um Okay. So like >> the the the thing you build with a >> the the the thing you build with a product like this, it it's still you product like this, it it's still you product like this, it it's still you could think of it as like a solution to could think of it as like a solution to could think of it as like a solution to a moment in time. You could think of it a moment in time. You could think of it a moment in time. You could think of it as a solution that allows people to do as a solution that allows people to do as a solution that allows people to do something repeatedly. um if it's just a something repeatedly. um if it's just a something repeatedly. um if it's just a onetime thing, don't build a tool for onetime thing, don't build a tool for onetime thing, don't build a tool for it, right? But but if it's like a it, right? But but if it's like a it, right? But but if it's like a repeatable thing that they have to do a repeatable thing that they have to do a repeatable thing that they have to do a lot of times, then you need to lot of times, then you need to lot of times, then you need to understand when that thing happens and understand when that thing happens and understand when that thing happens and then the so the iteration you do like then the so the iteration you do like then the so the iteration you do like the stepby-step thing is not about the the stepby-step thing is not about the the stepby-step thing is not about the moments in time in which they use it. moments in time in which they use it. moments in time in which they use it. It's about when they need it It's about when they need it It's about when they need it >> breaking down the specific steps like >> breaking down the specific steps like >> breaking down the specific steps like within the factory, it's like there's a within the factory, it's like there's a within the factory, it's like there's a conveyor belt and making sure you're conveyor belt and making sure you're conveyor belt and making sure you're QAing every step of the conveyor belt. QAing every step of the conveyor belt. QAing every step of the conveyor belt. uh I would say if the thing that you are uh I would say if the thing that you are uh I would say if the thing that you are thinking about productizing in this way thinking about productizing in this way thinking about productizing in this way does not lend itself to like linear does not lend itself to like linear does not lend itself to like linear progression progression progression >> then that way of approaching the >> then that way of approaching the >> then that way of approaching the development will be different like I I development will be different like I I development will be different like I I probably don't have as much advice for probably don't have as much advice for probably don't have as much advice for someone who's trying to build something someone who's trying to build something someone who's trying to build something that does not lend itself to like a that does not lend itself to like a that does not lend itself to like a linear progression because the way that linear progression because the way that linear progression because the way that the LLM's work is like every especially the LLM's work is like every especially the LLM's work is like every especially with interviews every step builds on the with interviews every step builds on the with interviews every step builds on the previous step. previous step. previous step. >> Mhm. So the solutions you come up with >> Mhm. So the solutions you come up with >> Mhm. So the solutions you come up with at the end of a product sense interview at the end of a product sense interview at the end of a product sense interview are built on top of the shoulders of the are built on top of the shoulders of the are built on top of the shoulders of the problems and the the segments etc. So problems and the the segments etc. So problems and the the segments etc. So you can't really just you know I've had you can't really just you know I've had you can't really just you know I've had people why can't I just get right into people why can't I just get right into people why can't I just get right into the solutions. I'm like because in in a the solutions. I'm like because in in a the solutions. I'm like because in in a real interview you wouldn't be able to real interview you wouldn't be able to real interview you wouldn't be able to get right into the solutions. You have get right into the solutions. You have get right into the solutions. You have to get you have to work your way towards to get you have to work your way towards to get you have to work your way towards that. that. that. >> Yeah I think so you mentioned two >> Yeah I think so you mentioned two >> Yeah I think so you mentioned two things. One the process is linear and things. One the process is linear and things. One the process is linear and the second thing is the process builds the second thing is the process builds the second thing is the process builds on the different steps. So it it it on the different steps. So it it it on the different steps. So it it it doesn't make any sense to either skip to doesn't make any sense to either skip to doesn't make any sense to either skip to a different level or to come back to a different level or to come back to a different level or to come back to another step. It does it's not circular. another step. It does it's not circular. another step. It does it's not circular. And the last thing I think is it doesn't And the last thing I think is it doesn't And the last thing I think is it doesn't diverge, right? It's like very focused. diverge, right? It's like very focused. diverge, right? It's like very focused. There is like a clear starting point and There is like a clear starting point and There is like a clear starting point and a clear end point, right? a clear end point, right? a clear end point, right? >> Yeah. >> Yeah. >> Yeah. >> Okay. I think that's really important. >> Okay. I think that's really important. >> Okay. I think that's really important. So we said okay understanding what goes So we said okay understanding what goes So we said okay understanding what goes looks like being able to actually build looks like being able to actually build looks like being able to actually build it bottom up and improve each step of it bottom up and improve each step of it bottom up and improve each step of the way understanding the user journey the way understanding the user journey the way understanding the user journey and exactly which point in time does and exactly which point in time does and exactly which point in time does this bring value and also mapping that this bring value and also mapping that this bring value and also mapping that within this step it's a linear process within this step it's a linear process within this step it's a linear process of progression right of progression right of progression right >> I think that's about right

>> I think that's about right >> I think that's about right >> okay um what else do you think is really >> okay um what else do you think is really >> okay um what else do you think is really important to to think about when you important to to think about when you important to to think about when you think about a co-pilot like this think about a co-pilot like this think about a co-pilot like this >> I think the scoping is a huge part of it >> I think the scoping is a huge part of it >> I think the scoping is a huge part of it >> um like like what we talked about >> um like like what we talked about >> um like like what we talked about earlier with like the the just trying to earlier with like the the just trying to earlier with like the the just trying to understand understand understand how much you're trying to bite off cuz how much you're trying to bite off cuz how much you're trying to bite off cuz you know I I I was teaching a course you know I I I was teaching a course you know I I I was teaching a course about both of these interviews but about both of these interviews but about both of these interviews but breaking them down into separate ones. breaking them down into separate ones. breaking them down into separate ones. We talked about that that's that was an We talked about that that's that was an We talked about that that's that was an important thing. Um, and then I would important thing. Um, and then I would important thing. Um, and then I would say like, you know, interviews say like, you know, interviews say like, you know, interviews kind of were a nice use case to get kind of were a nice use case to get kind of were a nice use case to get started with something like this because started with something like this because started with something like this because they have clear evaluation rubrics that, they have clear evaluation rubrics that, they have clear evaluation rubrics that, you know, you could help someone you know, you could help someone you know, you could help someone understand understand understand >> what good looks like or what weak looks >> what good looks like or what weak looks >> what good looks like or what weak looks like, like, like, but a lot of life doesn't have but a lot of life doesn't have but a lot of life doesn't have evaluation rubrics, evaluation rubrics, evaluation rubrics, >> right? So I think you know I I think um >> right? So I think you know I I think um >> right? So I think you know I I think um like great management in general like like great management in general like like great management in general like being clear about expectations and being being clear about expectations and being being clear about expectations and being clear with what looks what good looks clear with what looks what good looks clear with what looks what good looks like and what does what good does not like and what does what good does not like and what does what good does not look like or what bad looks like and look like or what bad looks like and look like or what bad looks like and being able to articulate that in words being able to articulate that in words being able to articulate that in words um I think will be very important too. um I think will be very important too. um I think will be very important too. um like you you can't I don't think you um like you you can't I don't think you um like you you can't I don't think you can get anything out of this kind of can get anything out of this kind of can get anything out of this kind of product unless you also take the time to product unless you also take the time to product unless you also take the time to kind of create some kind of create some kind of create some quantifiable way for it to evaluate its quantifiable way for it to evaluate its quantifiable way for it to evaluate its own performance because that's the other own performance because that's the other own performance because that's the other thing I would do at the end of the uh thing I would do at the end of the uh thing I would do at the end of the uh the co-pilot will also evaluate its own the co-pilot will also evaluate its own the co-pilot will also evaluate its own performance at the end of the performance at the end of the performance at the end of the interviews. interviews. interviews. >> Okay. So I think evaluation is another >> Okay. So I think evaluation is another >> Okay. So I think evaluation is another key principle is like the fact that key principle is like the fact that key principle is like the fact that there is some sort of feedback loop there is some sort of feedback loop there is some sort of feedback loop between iterations and I think that's between iterations and I think that's between iterations and I think that's critical, right? It's like you don't critical, right? It's like you don't critical, right? It's like you don't just do it once. As we said, if it's just do it once. As we said, if it's just do it once. As we said, if it's once, don't build the co-pilot. But you once, don't build the co-pilot. But you once, don't build the co-pilot. But you have to also when you finish that linear have to also when you finish that linear have to also when you finish that linear progression to have clear feedback loop progression to have clear feedback loop progression to have clear feedback loop into the entire process. >> Yes. Um and then and and the the other >> Yes. Um and then and and the the other reason that helps is because I knew reason that helps is because I knew reason that helps is because I knew there was actually two kind of workflows there was actually two kind of workflows there was actually two kind of workflows I wanted to bite off. One was how how do I wanted to bite off. One was how how do I wanted to bite off. One was how how do how do I work through a question I've how do I work through a question I've how do I work through a question I've never seen before? Like um never seen before? Like um never seen before? Like um >> so that was like the the main one, but >> so that was like the the main one, but >> so that was like the the main one, but then also I just filled out this then also I just filled out this then also I just filled out this template. Like what can you give me template. Like what can you give me template. Like what can you give me feedback on my filled out template? So feedback on my filled out template? So feedback on my filled out template? So to serve the second use case, I had to to serve the second use case, I had to to serve the second use case, I had to build the evaluation rubric. And then I build the evaluation rubric. And then I build the evaluation rubric. And then I realized that if I could solve the realized that if I could solve the realized that if I could solve the second use case with the evaluation second use case with the evaluation second use case with the evaluation rubric that the natural performance I rubric that the natural performance I rubric that the natural performance I got for the first case was got got for the first case was got got for the first case was got significantly better because I had significantly better because I had significantly better because I had codified what the evaluation rubric codified what the evaluation rubric codified what the evaluation rubric looked like. looked like. looked like. >> So um yeah I so my learning from that is >> So um yeah I so my learning from that is >> So um yeah I so my learning from that is you don't necessarily need to be you don't necessarily need to be you don't necessarily need to be building a product that has like a building a product that has like a building a product that has like a co-pilot that can evaluate some filled co-pilot that can evaluate some filled co-pilot that can evaluate some filled out template or something. But my out template or something. But my out template or something. But my learning from that was learning from that was learning from that was codifying the rubric for evaluation is codifying the rubric for evaluation is codifying the rubric for evaluation is something that would strengthen the main something that would strengthen the main something that would strengthen the main use case. use case. use case. >> Mhm. >> Mhm. >> Mhm. >> The main workflow. >> The main workflow. >> The main workflow. >> So Ben, we're about to wrap up and I >> So Ben, we're about to wrap up and I >> So Ben, we're about to wrap up and I would like to ask you one final question would like to ask you one final question would like to ask you one final question is like what would be like a key insight is like what would be like a key insight is like what would be like a key insight you want our listeners to stay with you want our listeners to stay with you want our listeners to stay with after hearing our uh conversation today?

I think that I think that people people people a lot of people dramatically under a lot of people dramatically under a lot of people dramatically under undervalue undervalue undervalue how unique their knowledge is about how unique their knowledge is about how unique their knowledge is about about specific things that they might be about specific things that they might be about specific things that they might be the experts on. And I think that in this the experts on. And I think that in this the experts on. And I think that in this new world we're going into, new world we're going into, new world we're going into, >> there's just like a lot of slop. Um, and >> there's just like a lot of slop. Um, and >> there's just like a lot of slop. Um, and there's a lot of like I think it's there's a lot of like I think it's there's a lot of like I think it's getting harder and harder to know what getting harder and harder to know what getting harder and harder to know what content to trust and what content is content to trust and what content is content to trust and what content is reliable. reliable. reliable. >> And so my hope is that by talking more >> And so my hope is that by talking more >> And so my hope is that by talking more about my co-pilot, I don't want people about my co-pilot, I don't want people about my co-pilot, I don't want people just being like, "Oh, that's so cool." just being like, "Oh, that's so cool." just being like, "Oh, that's so cool." Like I I mean, I think it's cool, but Like I I mean, I think it's cool, but Like I I mean, I think it's cool, but that's not the point of this. The point that's not the point of this. The point that's not the point of this. The point of this is I want people to be like, of this is I want people to be like, of this is I want people to be like, "There's nothing particularly special "There's nothing particularly special "There's nothing particularly special about me." Um, if I can do this, you can about me." Um, if I can do this, you can about me." Um, if I can do this, you can do it, too. And but I can't do the heavy do it, too. And but I can't do the heavy do it, too. And but I can't do the heavy lifting for for you as as a listener, as lifting for for you as as a listener, as lifting for for you as as a listener, as an audience member for what that thing an audience member for what that thing an audience member for what that thing should be and how you do it. That's like should be and how you do it. That's like should be and how you do it. That's like the hard part is obviously you could the hard part is obviously you could the hard part is obviously you could lead a horse to water, but you know, you lead a horse to water, but you know, you lead a horse to water, but you know, you can't make it drink. Um, and I so I but can't make it drink. Um, and I so I but can't make it drink. Um, and I so I but I do want people to kind of potentially I do want people to kind of potentially I do want people to kind of potentially consider building a new type of product consider building a new type of product consider building a new type of product that they hadn't considered before. And that they hadn't considered before. And that they hadn't considered before. And just because you could vibe code just because you could vibe code just because you could vibe code software products now doesn't mean that software products now doesn't mean that software products now doesn't mean that that's like the best way to maybe that's like the best way to maybe that's like the best way to maybe package and monetize your knowledge. package and monetize your knowledge. package and monetize your knowledge. Maybe there's like a faster path to Maybe there's like a faster path to Maybe there's like a faster path to getting people to what they need. And getting people to what they need. And getting people to what they need. And then as a consumer of your knowledge, then as a consumer of your knowledge, then as a consumer of your knowledge, like if I'm looking for expertise on, like if I'm looking for expertise on, like if I'm looking for expertise on, you know, how to improve my website or you know, how to improve my website or you know, how to improve my website or how to improve conversion in my funnel how to improve conversion in my funnel how to improve conversion in my funnel or how to better how to like these or how to better how to like these or how to better how to like these guides that we're doing with insider guides that we're doing with insider guides that we're doing with insider loops, if someone has like spikes and loops, if someone has like spikes and loops, if someone has like spikes and they're like the top 0.001% they're like the top 0.001% they're like the top 0.001% when it comes to like information when it comes to like information when it comes to like information architecture and like guides and like architecture and like guides and like architecture and like guides and like written guides and they could I could written guides and they could I could written guides and they could I could like buy something from them that just like buy something from them that just like buy something from them that just like allowed us to level up the level up like allowed us to level up the level up like allowed us to level up the level up our guides. Like there's so many things our guides. Like there's so many things our guides. Like there's so many things areas in my life and I'm sure for you areas in my life and I'm sure for you areas in my life and I'm sure for you too where you're looking for like what too where you're looking for like what too where you're looking for like what would the best person in the world do at would the best person in the world do at would the best person in the world do at this point with this thing and I just this point with this thing and I just this point with this thing and I just don't think there's like a place to go don't think there's like a place to go don't think there's like a place to go and discover a lot of that right now and and discover a lot of that right now and and discover a lot of that right now and I I would certainly be a buyer for very I I would certainly be a buyer for very I I would certainly be a buyer for very specific things that fit into very specific things that fit into very specific things that fit into very specific workflows for very specific specific workflows for very specific specific workflows for very specific expertise. So that's kind of I guess expertise. So that's kind of I guess expertise. So that's kind of I guess like maybe we're moving into a world of like maybe we're moving into a world of like maybe we're moving into a world of um kind of like micro micro expertise um kind of like micro micro expertise um kind of like micro micro expertise insertion points and that would be that insertion points and that would be that insertion points and that would be that would be really cool if someone chose to would be really cool if someone chose to would be really cool if someone chose to build something out of this. build something out of this. build something out of this. >> Amazing to me is like when we think >> Amazing to me is like when we think >> Amazing to me is like when we think about whether I want to scale myself and about whether I want to scale myself and about whether I want to scale myself and personalize the answers that's a good personalize the answers that's a good personalize the answers that's a good starting point but the main takeaway starting point but the main takeaway starting point but the main takeaway from my perspective is resist the urge from my perspective is resist the urge from my perspective is resist the urge of building this top down. So don't go of building this top down. So don't go of building this top down. So don't go and create tons of instructions but and create tons of instructions but and create tons of instructions but build this bottom up. start with the build this bottom up. start with the build this bottom up. start with the baseline and then add gradually more baseline and then add gradually more baseline and then add gradually more instructions and make sure that you're instructions and make sure that you're instructions and make sure that you're controlling the the output. And I think controlling the the output. And I think controlling the the output. And I think that's key to getting a result that that's key to getting a result that that's key to getting a result that would be gold standard and you can put would be gold standard and you can put would be gold standard and you can put your name on it. So Ben, thank you so your name on it. So Ben, thank you so your name on it. So Ben, thank you so much for uh this conversation. It was much for uh this conversation. It was much for uh this conversation. It was really really interesting. really really interesting. really really interesting. >> Yeah, thanks for having me. This was >> Yeah, thanks for having me. This was >> Yeah, thanks for having me. This was fun. fun. fun. >> And for everyone listening, thank you >> And for everyone listening, thank you >> And for everyone listening, thank you for listening. You can follow us on all for listening. You can follow us on all for listening. You can follow us on all of the podcast software to make sure of the podcast software to make sure of the podcast software to make sure that you know when a new episode comes that you know when a new episode comes that you know when a new episode comes in. And again, thank you so much, Ben, in. And again, thank you so much, Ben, in. And again, thank you so much, Ben, for your time. And thank you everyone for your time. And thank you everyone for your time. And thank you everyone for listening.