גלו את כוח העל שלכם כמנהלי מוצר: סוד ה-AI קו-פיילוט שאתם חייבים להכיר!

Hi everyone, I'm Ran Arez and this is the podcast where we talk to different product managers about the problems they're facing, how they solved it, and what they learned along the way. And today we have a very important podcast episode. Today we are hosting Ben Arez to talk about an AI co-pilot experience and he's going to share with us his challenges and his learning and his trade-offs. So Ben, thank you so much for uh being here. >> Yeah, thanks for having me. This is fun. >> Amazing. So, for everyone who may not know you, maybe just introduce yourself a bit. >> Sure. Uh, so yeah, my name is Ben. I live in uh Brooklyn, New York with my my wife and our three-year-old daughter and uh 8-year-old golden retriever. Uh, we moved here a little over four years ago after 8 years in San Francisco. And before that, uh, grew up between the US and Israel. Went to school in Florida. Um, yeah, I started my career in finance. I got out very early and then I was a failed founder, but learned a lot and found my way into product management. Uh, joined a company called Life 360. uh as their 35th employee hired by another Israeli uh product guy named Idomar Novik and then discovered product management there and I'd say the rest is uh kind of history. I ended up spending uh the last the the following 10 years across uh series A B C startups. I also uh was at Facebook for a period which kind of like one of the most influential experiences that led to what we're going to talk about today. Um, and uh, I'd consider myself at my at my core like a zero to one product person. And a lot of the work that I've done in larger companies, um, you know, while while potentially more impactful on the on the scoreboard, I think still, uh, in my heart never I've always just loved being super hands-on with building things, talking to users, and just creating creating new things.

>> Amazing. So, let's before we dive in, let's just set the stage. What are we going to talk about today? Why is it important to other product managers? Yeah. So, I kind of alluded this to this in the intro, but the last couple years have been uh kind of like an entirely different chapter of my career. So, until two years ago, I was in full-time PM work. As I mentioned, I was the first PM at three different startups. I was a PM at Facebook, attentive, um I was a founder. And then two years ago, um I decided to do something different uh with my career and I kind of went off off the the beaten trail and started um what I might consider as a solarreneurship journey. Started off thinking about doing fractional product work like advising, consulting uh and I did I have done some of that over the last couple years but I would say the thing that really pulled me uh more than anything else that I've done over the last two years is uh product management interview preparation. Um I used to do this as a favor to people to help them get ready for interviews at places like Meta. And then at some point I helped someone land a very lucrative um offer after spending about six hours helping them prepare. Wow. >> And they're like, "You're an idiot for not charging money for this." So um I ended up starting to charge and um and then that all led to a Maven course that I launched about a year and a half ago, coming up on I guess like two years in in April. >> Wow. Um, and that Maven course paved the

path to my co-pilot, which is what we're going to talk about today, as a tool for helping my students, but ultimately that the impact of that expanded well outside of my course. And my niche, uh, I would say has really become product sense and analytical thinking interview preparation like the ones that are used at companies like Meta. I think you guys at Monday um, we're doing that as well. Lots of other companies use those interviews, too. And then um I have a few other things I'm spending my time on these days, but I think you know it's all it's all in the general PM interview prep space. >> Yeah. Great. So you said throughout the lines like co-pilot. How do you define a co-pilot? >> Yeah. I mean I think it's kind of um it's one of those words that has a lot of different meanings to a lot of different people kind of like agent. Um so I would say >> um let's just talk about what talk about it as a tool >> and then you know I can talk to you I can tell you why I decided to call it a co-pilot but but what the tool does at its core is you could think of people um using claude chatpt Gemini whichever LLM is their tool of choice and a couple years ago um actually you know what it was probably just a year and a half ago or so we had Tal Raviv on on my podcast and I got to know Tal >> and Tal was kind of just like telling everyone about Claude Projects and I'd never played with Claude Projects before. Um, >> but I started playing with it and I had a light bulb moment go off and what the light bulb moment was is I had at that point been teaching my course for about five or six months and I started to get a really good feel for what people were asking me about in my course. They were asking me for help filling out my templates. I did some um I created these very popular and they're free like interview templates for product sense and analytical thinking interviews and people would make a copy and start working through a question from a question bank and then they would fill it out and they would ask me hey like could you give me feedback on how this submission is um and that would be very timeconuming because it wasn't like a 10-second thing. It was like I had to understand the whole context of the exercise. So I was like I need I need a way to scale my ability to give feedback on those filled out submissions. And then the other problem was some people get very fixated on the way a question gets asked in the question bank. So like they see a a flavor of a of a way that an interviewer asked a question they've never seen before and it kind of like can be kind of scary for them. >> So they're like how would you tackle this? Like how does this come back into your framework? So I found myself spending a lot of time help you know getting on calls with people and being like here's exactly how you kind of fit this into my framework. Don't be intimidated by the question. So I had these two big problems and I was just spending like way too much time with students trying to answer that and I knew that wasn't going to scale. So what I started tinkering with in Clawude projects is a way to basically codify my framework for how to answer these questions. Give it my template so it's aware of how the template works and then give it some examples of how filled out templates might look or how filled out submissions might work and then give it a very clear evaluation rubric to start. And then I just like that was kind of like the in that was like my light bulb moment for the co-pilot. I end up calling it the co-pilot because what it is is it's it's like a practice co-pilot when I tell people you're going to spend 20 30 hours maybe getting ready for your meta interviews or you know your product sense analytical thinking interviews at another company. Um how are you going to use that time? Are you going to spend it just doing work on your own? Are you going to spend that all on mocks? Are you going to spend it doing something else? Like what are you going to use that time for? So, I tried to create a very opinionated path for how people could spend that time. And I generally tell people to start with a co-pilot when they're ready to get reps with the material, but before they do mock interviews. >> I think when we think about other PMs listening to this is we try to frame it as either a challenge of how do you scale yourself and how do you personalize your answers when you're not there. And when we think about this framework of personalization and scale,

I think this is the initial questions when we need to ask when we could think about maybe a co-pilot is right for me at the situation. I think today what we're going to do is we're going to break down both through your use case but also to expand it to other let's call it Ben's playbook for a co-pilot to really understand the nitty-gritty and everything that we need to think about when we think about launching a co-pilot to either scale ourselves, personalize ourel or both of them. Okay, sounds good. Yeah, let's do it. >> Okay. So, let let's take your case study and and and let's let's start. You said it was about a year ago when you started it. Maybe walk us through the steps you took to to get it off the ground. >> Yeah. I would say the very first thing

was just figuring out if it even works, >> right? So, I wasn't thinking about marketing. I wasn't thinking about packaging or pricing. Um the original idea was I was wondering if I could build something that helped me um do something as like an internal tool as an instructor. >> Okay. So deep understanding of my own workflow um was kind of like the foundation and obviously having a deep set of like pretty strong opinions about what good and like not good looks like. >> Um I'm we're going to be trying to generalize this live for the first time. Okay. So I I think some of this is going to be a bit you know raw as we talk about it. But maybe if the audience has any um uh kind of like light bulb moments as we talk through this for them learning how to apply this to their life, I actually would really appreciate hearing that from them because I'm I'm on the lookout for more ways that people are doing this kind of stuff because I don't think there's anything particularly special about me. But I don't think I've also codified this into like a clear playbook that just like anyone can do immediately. >> Yeah. So it's a work in progress. That's okay. I think >> yeah it is a work in progress and I would say like so that that was kind of like the first moment is like trying to see if I could work through a single question with my with this project and I started with a product sense question. >> Uh in hindsight I probably would have started with analytical thinking. I think it's actually kind of a little easier to codify um analytical thinking interviews in this way than product sense but um but nonetheless that's the first step is just play with it for yourself. build something that you could use that adds value to you and gives you leverage for something that you find yourself doing a lot. >> Okay. Okay. >> So, and that was the point where you thought about at first it was just internal just for you to get more leverage as an instructor, right? >> So, you had the need of actually answering more and more people and that's where you started working through the product sensory, right? >> Correct. Um there's there was that and there was also kind of like a business need. You know, as like every good product person, you're trying to solve a human problem and also trying to solve a business problem. And the business problem was I couldn't see a path for myself to scale my course beyond maybe like 10 or 15 students at the current at the amount of time I was spending with my students. So I had a business need that's like if this is going to be able to grow, I need I need some systems that allow me to grow this. Mhm. >> Um, and then I think there was another business need slash a uh pricing and positioning need which was I wanted to be able to justif to command a more premium price for my course. And I was just like really strongly thinking about what is the most valuable thing I could add to the course beyond what I'm already doing. And some kind of tool that people could use to to do their own practice that was like a proprietary tool felt to me like it would be very value ad. Mhm. >> So when all those conditions were right, that kind of made me start really playing with this and getting excited. >> Okay. But but the first thing is I think first of all we need to see demand where you try to repeat yourself over and over again in a specific area and then you say okay I have a market pool of actually questions coming back and they usually are similar questions but on maybe slightly different ways of asking the same thing. Right? I think that's the first thing of actually seeing a general demand that's very similar. Right? It's not like questions that are very very different, right? >> Uh totally agree. And and and it's clear in hindsight that there were two buckets of questions I was getting asked, but in the moment it wasn't obvious. Like in the moment I just felt overwhelmed, right? >> Like I'm just like, "Oh my god, I just ran this course. I I ran my first cohort in July and then I took three months and I ran the next cohort in October and then it was around um the holiday season, like end of December. I was getting ready for my January cohort and I was kind of like, I don't know how this is going to go. I'm really worried about that, but I definitely feel overwhelmed. So, I just started kind of chipping away at the problem. In hindsight, >> it's clear to me that my attempt to chip away at the problem was >> had implicitly prioritized, let's say, a couple big buckets of questions that I got asked. >> Okay. And then so we see the demand and we see that we can start scaling ourselves >> through this. Then you started working on uh cloud projects right? >> Yeah. >> Okay. What was that uh looking like is in in in the real real time what did you do? >> Yeah. So uh the cool thing about cloud about I guess like AI tools at that point cloud projects at that point is there weren't that many um levers to play with >> right? So you had instruct you had you create a project and then there's like instructions and there's files. >> Okay. So >> that's basically all it was and it was doing rag um to kind of try to come up with answers. Um and so uh the first thing I tried to do is I just didn't put in any files. I didn't put in any instructions. I just asked an interview question. >> Okay. So, I said, you know, um, how would you I think I might have asked like the most common question that I've seen over the years, which is something like, how would you build a product for volunteering? >> Yeah. >> Yeah. >> Right. I I may have said you're a PM at Meta. Um, how would you build a product for volunteering? Or something like that. Um, and I just kind of saw what it came up with and obviously it jumped right into the solution >> solution, right? >> Like a bad PM, right? >> Like a bad PM. So I was kind of like, okay, like I guess if I just it would have no it would have no way of knowing that that's not the right way to do to do this. So, what if I go in the instructions um and I say, "Hey, like every chat in this project is going to be me asking you an interview question and before you jump into solutions, I just want you to frame the context around why this matters and then, you know, walk me through who you might build this for and then tell me what their problems might be and then and then I want you to land on a solution at the end." Mhm. >> So, I just started very simple like that and then I went back to the chat and Claude Projects has this button called retry >> and you could hit retry with no changes and what that does is it basically has the the the AI take another stab at generating the response but it does pull the latest kind of instructions from the project >> and then I got to see what it did you know with the latest instructions and then I was like okay that's a little bit better but this part is off and that I basically just did that like a few hundred times over the course of probably a couple weeks. >> Mhm. >> Just to kind of get the output from the response for for product sense and analytical thinking to be what I would kind of expect to see in a in a passing

answer. >> So I want to I want to highlight two things that I think you did which are really really important is the first thing is seeing the baseline. What do we do without any instructions? Like you saw the baseline case and you saw that you could improve that dramatically or at least you had an hypothesis. And I think this is really crucial because a lot of times as PMs we run into solutions as well and we try to create more instructions where maybe the best the base case is okay. And I think when you saw that it runs to solutions like we do as PM sometimes I think that's that's a great signal to start as a baseline. The second thing is instead of creating all of the instructions at once and seeing the results you built it like bottom up. You started with the baseline and then you added one instruction and another one and another one and you actually control the the output with a lot more granularity than what we I see sometimes is PMs throwing a lot of instructions and having no idea how a word might change their results. And I think the last thing that I want to highlight is the fact that you retried but without the context. So it's like a clean environment that you started from fresh every time. So you didn't have like a dirty environment of context in the AI to actually learn from their mistakes and not from your instructions, right? >> Yeah. And then when I did feel like the environment was getting contaminated by the context of a given chat thread, I would just start a new chat and be like, "Okay, now let's start from scratch." Like it has none of that context. And I did just like lots and lots of threads. But yeah, you're right. I what what I was trying to do is I was trying to kind of get a feel for this machine, >> right? >> What is this thing? How does it work? like how um how much of this kind of like can I really nudge in the direction that I want to nudge it versus how much of it regardless of nudging is just not going to quite you know it's just going to keep dancing around the thing but it's not going to quite get there. >> Yeah. >> And um >> And um >> I found that like if you think about the what what I was just blown away with is the adherence to like the overall instructions and structure. M >> like I could not like cuz in a lot of ways that is really the hallmark of the best candidates at a company like Meta. It's like their ability to do the expectation setting up front to like set the right expectations to lay out their game plan for the interview and then to like actually hold themselves accountable and to stay on on on the path as they go through the interview and to way point along the way and check in with you. >> Like that was the thing that I kind of was most blown away with initially. It was less like the the the quality of the content that that it was generating, but it was its ability to get through the whole exercise by playing the game that I asked it to play >> and internalizing my framework down to like how it evaluates problems by scoring them on, you know, severity and frequency or picking um a segment based on reach and impact um you know, like or I'm sorry, reach in underserved degree relative to the mission. Um, so it's like just ability to understand the game I wanted it to play was by far the most impressive thing that I kind of made me get excited less it felt like the content itself was more of like um implementation detail that could be figured out later like getting the content to be better. >> So one of these light bulb moments was that the instructions were like the best case scenario of following the instructions. Now we need to tweak each and every step maybe to generate more like nuanced answers. But you saw it like going through to the letter of what you said is important as a meta interviewer. >> Yeah. Like and and I can't un this is one of those things that like people just get wrong in general even with human coaching is like when I tell them how to get ready for interviews. Some people just um they completely discount the value of having structure >> in interviews. And so the fact that I could get the the the the co-pilot, the the AI to do it like pretty early on was to me like a very strong signal because I just needed people to remember it's like there's no such thing as too much repetition when you're coaching someone. >> I just needed them like to know every question you work on, I do not want you to skip this step. Always start with assumptions. Always walk through your game plan and then get into the meat of it. >> Amazing. Okay. So now you have something sort of working if I understand correctly. you did like hundreds of iterations around that but you managed to get it through an interview process end to end. >> Um well the first very first version I

actually had one kind of project for both product sense and analytical thinking I thought I could I thought that was like why not >> um and then I I I pretty early on as part of that initial tinkering phase at some point >> by I I I branched them out into two separate projects. So, one project for product sense and one project for analytical thinking because I realized while the game is similar, >> the fact that there's different evaluation rubrics at the end that it should be holding itself accountable to for each of them made me realize that having two different kind of potentially um conflicting evaluation rubrics would would kind of pollute the the quality of the experience. So yeah, I would say like to answer your question, once I got out of that initial tinkering phase, I had two projects, >> one for product sense, one for analytical thinking and then the the once I kind of was like I think this could potentially pass an interview like I feel like cuz by the way the the big decision we haven't made yet is like I had the AI play the role of the co-pilot, not the I'm sorry, the AI play the role of the candidate, not the interviewer. And the reason for that is because I found myself in the shoes of a coach and in the shoes of an interviewer doing mock interviews with people all the time. >> So I wanted like the thing I was most kind of calibrated to like insensitized to is what is the performance of a candidate look like? >> And so I wanted the AI to play the role of a candidate by really and then I wanted people to be able to use the AI to >> to see what a good candidate behaves like. Um, so once I had that all up and running, I pinged a few um, people that I knew who were calibrated interviewers from Meta. >> I'm like, "Hey, I built this thing for my course. I'm about to give it to my students. I don't want to take them off track. Um, and lead them astray. Could you just like ask it a question? Ask it this question, right? Or or pick any product sense question that you like to ask in interviews. Um, and just like be play the role of the interviewer and just if you agree with what it's saying, say yes every time it checks in with you. And if you think it's not making sense, ask it a question like you would ask any other candidate. Can you help me understand why you picked this segment or I'm not really following your thinking on that mission statement or can you better you could you help me understand how this connects to the company's goals? Like you know, like whatever questions you'd ask in a real interview, ask it. Um, and then just like get let me know when you're done and and let me know kind of like your general assessment of, you know, like how was that? >> Um, and one of the first responses I got

I got back was uh in and in in slightly different word. I'm not going to say who it was, but it was uh someone who was definitely someone I worked with at Meta, who was a calibrated interviewer, is not at Meta anymore. Um, he was like, "Hey, like can I hire this co-pilot? Like I feel like this this would be like a passing score for me." >> Wow. Okay. Yeah, >> that that's incredible actually. I think like um >> it's and I think it's down to both you understanding what great looks like and also being able to actually calibrate it step by step to actually make it look great. And I think the separation you said I think it's very important to say like once you you branched out into two different things is exactly what I think when I'm saying okay if it's a completely different instruction process or step-by-step guide then this could be a good signal to think about training them differently. It's like trying to have a sales guy also do marketing. Okay. It's like two different use cases essentially. >> Well, that one u not to get pulled into too big of a tangent, but you know, some companies obviously see sales and marketing as needing to >> be woven out of the same fabric and play kind of like a consistent game, use the same voice and um and you know that's why they rolled them sometimes into like a chief revenue officer kind of role is because they don't want them to function as two completely different things. But um but but I think your point stands is like I think if there is a specific uh workflow or a specific context space that someone finds themselves in and you don't want that context space to get polluted with a different context space than being really intentional with I mean this is kind of like the modern version of scoping in a way like it's you're kind of scoping the environments in which things can happen non-deterministically. >> Yeah. But like you're creating the boundaries and when you're setting the boundaries quite tightly, I think I think that's a good signal to actually train on the on the process. Even though you can move within the boundaries left and right and you can do segmentation differently maybe, but it's still around the same framework. And I think that that's key. >> Yeah. But but and and importantly like the way that you do segmentation in a product sense interview is kind of at a different level of altitude and fidelity than the the way you do segmentation in analytical thinking interview. um like analytical thinking I want to I want you to stay at the level of high high level ecosystem players right it could it could even be as broad as like the supply and the demand side of the marketplace >> um but in a product sense interview that's kind of like step you know step one >> and then you want to kind of show me that you could identify >> several ecosystem players not just like the the key supply and demand side but I want to really understand your ecosystem thinking and then I want to see how um how you could break down how you could pick one of those ecosystem players and then use something other than just like boring demographic kind of breakdown segmentation heristuristics to craft unique segments >> that you could then put into some kind of persona and then brainstorm some problems for that persona along a journey and then land on some solutions and pick one and tell me about that right and so so like there and what I tell people in my course all the time is like analytical thinking and product sense it's almost like you're parking at the trail head. You're going to go on a on a hike in nature. Like you're hanging out. You're starting at the same point. You're going to start with some assumptions and set some rationale or motivation for why this matters, but then they're going to go in different directions the moment you start to kind of get into the step after that. >> Yeah. And I think Okay, so I'm going to back to the story. So you gave it to a few ex meta interviewers or meta interviewers. You got calibrated feedback saying, "Can I hire this person?" Which is amazing signal. Then what did you do? Yeah, it's also, I guess, like a testament to the the shortcomings of modern hiring practices, which is another thing we could talk about. Um, but you know, hey, if they want to play a game, like candidates are just need to learn to play the if the company is going to have candidates play a game, then you can't blame the candidates for trying to get really good at playing that game. >> Um, so what happened after that is I uh gave it so I had like I mentioned to you, I had already run two cohorts. >> Mh. >> Okay. So what I did is I before I started my January cohort, because this was all happening again over the course of like a two-eek period. So before I put it in the hands of my January cohort students, which was like a couple weeks out, I emailed all of my my quoteunquote alumni who took my my July and my October cohorts, and I said, "Hey, like this is free for you. I'm just giving this to you. It's like I I took all your feedback. I listened to all the questions you had. I'm thinking ahead to my next cohort. Some of you are still in interview prep. Some of you already landed offers. That's cool. But like regardless, it would be it would mean a lot to me if you could just like play with this and tell me how helpful is this? What what could be better? Um, so that's kind of like wave two is like putting in the hands of the alumni uh from the course and getting their feedback. So I think here if I highlight two things that are important is first of all you gave it to the interviewers essentially to make sure that the bar is high enough and then you gave it to the actual let's call it the persona who is going to actually use it to actually use it and see how this goes and to get that essential feel for whether this gives personalization at scale. Right. >> Okay. Totally. >> And then and then what did your alumni say? Um, I mean, like it was it was very positive, but I'm I'm trying to kind of frame this in a way that doesn't sound like I was just like I I genuinely was looking for what's wrong with it. >> And I did get like the overwhelming kind of feedback I got is like this is incredibly helpful. >> Wow. and >> Wow. and the like I felt like the um you know the segmentation just felt a a little kind of like shallow to me or it didn't really follow like what you said or it picked the supply side of the ecosystem when I I think you generally recommend for this kind of question maybe to pick the demand side >> um cuz you know for like a consumer interview I generally recommend going deep in a product sense on like the demand side or the consumer side >> um and I was like okay yeah I never told the instructions that it should do that >> um or when it got into problem identification It didn't do like uh people were like, "Yeah, you always talk about how you should try to do a journey before you pick a problem." But the co-pilot wasn't doing a journey. It was just kind of going from like, you know, um a segment to like the problems. >> And so I I I actually want to get more practice with the journey piece, um etc. So I was I was starting to get kind of like specific pieces of feedback uh or like the mission statements were too verbose. M um so people were kind of like holding me accountable to like the stuff that I told them and they were like look like >> I'm confused I'm confused because you

just gave me this tool and like I'm trying to figure out whether what you told me is the source of truth or what the co-pilot is telling me is the source of truth. >> And that created an urgency for me to be like like I can't put this I can't like >> give this now to a bunch of new people in January if I'm going to contradict myself like if there's going to be contradictions in the system. Mhm. >> So that's when I kind of went into like intense tinkering mode for like phase two of tinkering, which is like, okay, now I'm going to literally go section by section in both of these. I'm going to look at the last five questions in the question bank from Louiswis Lynn. I'm going to drop them into new chats and I'm not going to basically like be done with this process until like until until there's no contradictions between what the co-pilot does and what I would do. And I'd say in the first phase I was a little bit more forgiving of it because I was like okay like some people might be okay >> with like >> with like >> it's just a co-pilot >> for me it's just like this this tool but like when people actually started being like I'm confused and I'm actually like I'm a little less confident in what I should do for this step because of this. >> I was like okay that's the stakes just went up and I need to like take up I I need to take the instructions way more seriously. >> Yeah. I think I think essentially at this point if you haven't done anything else it would detract the value of you as a coach essentially. It's like can I put my name on it? It's like first of all, okay, I can put it in front of interviewers. Okay, got a good thing, good signal, but can I really put my name on it? And I think that's that's a key moment in time where you say, okay, now it become it's ready to be productized essentially where I can put my name on it, right? And that that was think that that was the point you had. >> Yeah, for sure. And um and look, I I take I take like being helpful to people like I can't I can't I can't overstate how how important it is to me that um people walk away more confident and more clear on what they need to do after working with me or taking my course. And so, you know, um if I had like a neutral impact on people, they're like, "Yeah, it's like, you know, whatever." Like I was kind of I would take that one way. And if they were just like, "It's perfect," that's one thing. But if but whenever someone tells me that they're um that they're more confused because of something I gave them, that's where I'm kind of like I can't put my name on that. Like that's that's like a quality bar that I hold for myself is like I I refuse to like put things out there that will hurt people's preparation in any way, which sounds obvious, but like >> I let's just say I don't think that's like the I I think I might have a different bar for that than some other people in the space. >> Mhm. But you didn't stop there, right? So even after you tweaked it a lot and did a lot of a lot of feedback after your students feedback, you continue to improve it, right? You had the ultimate test of making sure the co-pilot works. >> Yeah. And and I'd say right around the

time where I was like comfortable giving it to my January students. That's when I felt comfortable increasing the price of the course >> to reflect to like actually position put co-pilot in the name. >> The reason I called it co-pilot is because Tal Raviv was marketing his copilot stuff at the time. I talked to Tal I talked to Tal about it and Tal was like just just like put C-Pilot in the title. >> Um >> and um and then so that allowed me to go from like $600 to $1,000 for the course price. >> And I said, "Okay, like um what if someone doesn't want to take my course, but they want to get access to this tool, do I want to let people get access to it?" M >> um and I was like, "Okay, I I'll sell it separately and then I'll just bundle it in the course." >> But for me to sell it separately, to your point, I did have to be that was like my first time ever selling a digital product like this. So, um I didn't expect anyone to buy it. I thought I I just But then it started to happen because I linked to it from the the the Maven course page >> because I was like, "Hey, like this thing is included in the price, right?" So people would go and check it out. Maybe they didn't want the course, but they bought the co-pilot. Um, and I told people if they buy the co-pilot, then I'll just give them a discount in the amount of the co-pilot, which the the original price was $2.99. So I basically like if you buy it, I'll give you a $300 promo code on the course. So you'll you'll be made whole if you end up wanting the whole thing after. >> Yeah. So you you don't have anything to lose if you decide to go in this route. You could just try it out. And I think uh I think that's that's a great signal where at the end of the day, someone was paying for that advice. And did you get any refunds at that point? I mean, I think I've only ever had two >> co-pilot refunds over hundreds of sales at this point. >> Um, the course has refunds, but every instructor will >> tell you about that. I think there's like a lot of students that will just sign up for a course not reading the page, not knowing what it is. They see a guarantee from Maven, and they'll just sign up, not thinking twice about it, and then, >> you know, they'll just tell themselves a story about why that was a bad purchase. they'll get cold feet and they'll want their money back. But it's it's never because like I'm not delivering what I said I was delivering. It was more because they had some story in their head about what it would be. >> And instructors generally see somewhere between like 5 to 10% refund rates on their their Maven courses. >> Okay. So when when was the moment that like this co-pilot exploded? >> It was probably April when I had my April 2025 when I had my um my Lenny guest post go live about Product Sense interviews. because that linked to all my things that linked to my course that linked to my templates that linked to my co-pilot um and uh that also basically established my number one spot on like you know SEO if someone looks for product sense interview and my my my stuff just pops up at the top. >> Um >> yeah, I think I did like $10,000 of co-pilot sales that that April. Did did you did you use the co-pilot when you wrote the Lennier articles? >> Uh for the >> I did I would say I did it much better for the sec for the analytical thinking one. >> Um again I think the analytical thinking stuff I'm just generally more proud of what the co-pilot does with analytical thinking. Um, so I felt more comfortable, I don't say delegating, but really like >> we we we like went big on examples in the analytical thinking post. Basically did one example. >> It's a great >> but then linked to two separate examples um in the in in sheets and the content of the sheets that were populated for those were all generated with my co-pilot. >> Um, for the product sense one >> I did generate the content with my co-pilot but I did a lot more kind of like iteration and tweaking. uh to to kind of get it ready to go live. But the I think the thing I was telling you when we were kind of doing our uh preparation for the episode is um when when Lenny and I started working on those posts, I I it wasn't like to try to save myself time that I wanted to leverage the co-pilots, but it's more like I wanted to use my co-pilot I wanted to use the Lenny post as a forcing function >> to make my co-pilots better because I expected a lot more people to come and buy them. So there was >> a lot more people to come to my course. >> So essentially there's like this is the golden standard. If I can make this here as a forcing function, I think you'd be comfortable selling it, people using it because you use that for what's considered the golden standard and if you are proud of it and not just putting your name on it, but really proud of it being out there and I think that's a great signal. So >> yeah, I think Lenny Lenny has arguably like the most premium editorial voice of anyone that that I'm aware of that does like in our space. So yeah, my my idea was if I'd be comfortable putting my name under this uh behind this and like it would go live on Lenny's, then it would be good enough for like basically everyone else to use it.

>> Amazing. So, we touched a lot of things about your case study, but I want to help our listeners maybe think about like Ben's playbook for building a co-pilot. And I want us to try and think about like the main principles that we talked about here and try to help our listeners think about their problems and whether a co-pilot is good or bad for this situation. So, if if I had to ask you, what would you say would be like the biggest principles you see? I would say the first one is to like really sensitize yourself to what you find what what kind of advice or what kind of um expertise you find yourself using the most >> where >> where >> you would be comfortable saying I might be like the best person at explaining this thing. Okay. Or there's other great people at explaining this thing but for some reason the way I explain it or the way I you know tackle it seems to really resonate with people. So trying to like really sensitize yourself to where you might have a superpower of some kind. Okay. >> Um and the reason that's important is because everything we talked about relied on my ability to evaluate the quality of the output. >> Yeah. >> Okay. And if I was trying to do something that I was not an expert on or that I wasn't extremely opinionated with real proof points that my opinionated beliefs were backed by real results, um then I would be uh shooting in the dark or I would have no ground to stand on as like a credible voice. And if you're just trying to build something for like for fun or just like you know as a hobbyist or you're not trying to like take a like a high stakes thing like interview preparation which is super high stakes then you know maybe you could like loosen the the requirements around that a bit. But if you if you are trying to bring your expertise into some kind of packaging that um people would pay for and maybe even command a premium price for then I would really try to like narrow in on that thing that you're uniquely positioned to do. And then um I would like we said establish a baseline. >> Okay. The re why use clawed projects

versus build software or you know in 2026 right? >> Yeah. Um, >> Yeah. Um, I would still argue that, okay, I did spend about four or five months last year with a really good engineer friend of mine building kind of like our own, >> you know, GPT wrapper, um, or our own, >> think of it like, you know, um, a Claude project software product that was using API token keys. Um, and we just started playing with it to get a feel for like, hey, like what if we could fully own this experience and build like a co-pilot platform, right? I was like, maybe that's the direction this all goes as I build a co-pilot platform. Um, and let me just tell everyone right now, like the amount of money that you're saving by using Claude's products or Chat GPT's products or Gemini's products, like the the sheer subsidies for tokens that we get as consumers is unimaginable until you start playing with these things with your own product. >> Yeah. >> Yeah. >> So, that direction died very fast for me. Um, so that's kind of why I I wouldn't recommend necessarily building a new product for for your thing. And I do see people build their own co-pilots and their own stuff and >> because it's cool. >> It's cool. It's fun, but like but it's completely the wrong ide. I think uh for us to try building it like like that. like like that. >> Yeah. And and like I'll take I'll take it even a step further. Like the better the product and the better the co-pilot, the more usage it's going to get, which means the more token heavy the consumption will be. >> Yeah. >> And if people come to if you're like, oh, like we're not we're going to build our own because like the chats won't even be that long or we're not going to even need that many tokens. It's like, well, does that mean like people don't actually value like are you basically building something so useless that people don't want to use it that much? Um, so the usage I was getting, I was regularly hitting usage limits in claude projects before they had their $200 tier. So I I would have to like stop in the middle of an interview practice, like an session, >> and I'd have to come back like four hours later and like continue my tinkering because I couldn't get it to generate more results. >> Um, so that that was the kind of like heaviness of usage that I was getting in my tinkering. And then my students were like, "Hey, like how can I get around these usage limits?" And I'm like, okay, if they're hitting the usage limits, that means that they're really playing with this. Yeah. Right. >> Um, so for all those reasons, I would say like if you're just getting started, use like the out of the box things that you can get from Claude or even like a custom GPT. Although I have I don't think the custom GPT exp I I do offer custom GPTs to people that buy the co-pilot, but I generally point people in the direction of claude projects or or gems. I think those are probably the two best best ways to do this. But just start giving it just like where we started like let's say you want to um let's pick on uh running a user research interview or something. >> Okay. Something I'm also like >> when I was thinking about what to teach like it was actually between teaching a course about user research and talking to customers and like interview prep. So like I I have a lot of opinions about user research. Okay. Um but let's say I wanted to build a co-pilot for user research and I wanted to productize that. I would start by, you know, asking um asking the co the the AI to play the role of a um I would tell it who who the AI is. I'd be like, you're this person. Here's your job. Here's where you live. >> Yeah. >> I I would I would make sure it knew knew who it was >> and then I would be like, I'm about to do a user research interview with you >> and I want to um I want you to answer these questions as if you're a >> participant. So I'd say like once you know what that thing is, I would I would try to break it down into like what's the >> what's the workflow in which that expertise fits. >> What's so for interview prep, it's generally like, you know, someone's getting ready for an interview. They may take my course, they may not take my course. They're trying to get ready. There's probably going to be mocks at some point. So where where in that process does this thing fit?

>> Um and where does your tool fit? Do you want them to use use it during an activity, before an activity, after an activity? So, trying to really be opinionated about how it might like when does it fit? >> Um, yeah, I would say that's probably step two. >> Okay. So, step number one is like understanding what good looks like and the fact that the results that you're bringing as a human being actually produces great results in the real world and that you know how to determine whether this is a bad answer or a good answer. That's step number one. Step number two is actually mapping the user journey of your users and understanding where exactly in this process does this fit like is there is this a specific step before during or after and like really nail down like the step that you need co-pilot or you could use copilot right >> you know uh I'm realizing as we're talking about this like we're basically like talking about my product sense framework right like that's you're you got to figure out what the thing is what you know who is it for what problem are you trying to solve for them in in the context of what they do and then what's the what's the angle in which you want to solve that problem. That's basically what we're talking about, right? That that is the playbook. >> Yeah. >> Um so yeah. >> Yeah. But I think another thing that you that that came up during the case study is the fact that you could build it bottom up, step by step. I think that's a really important thing is that you can target a specific step within the process and actually improve it. actually tweak it until you are satisfied with this specific step and then move on. Right. >> Yes. But let me clarify what um Okay. So like >> the the the thing you build with a product like this, it it's still you could think of it as like a solution to a moment in time. You could think of it as a solution that allows people to do something repeatedly. um if it's just a onetime thing, don't build a tool for it, right? But but if it's like a repeatable thing that they have to do a lot of times, then you need to understand when that thing happens and then the so the iteration you do like the stepby-step thing is not about the moments in time in which they use it. It's about when they need it >> breaking down the specific steps like within the factory, it's like there's a conveyor belt and making sure you're QAing every step of the conveyor belt. uh I would say if the thing that you are thinking about productizing in this way does not lend itself to like linear progression >> then that way of approaching the development will be different like I I probably don't have as much advice for someone who's trying to build something that does not lend itself to like a linear progression because the way that the LLM's work is like every especially with interviews every step builds on the previous step. >> Mhm. So the solutions you come up with at the end of a product sense interview are built on top of the shoulders of the problems and the the segments etc. So you can't really just you know I've had people why can't I just get right into the solutions. I'm like because in in a real interview you wouldn't be able to get right into the solutions. You have to get you have to work your way towards that. >> Yeah I think so you mentioned two things. One the process is linear and the second thing is the process builds on the different steps. So it it it doesn't make any sense to either skip to a different level or to come back to another step. It does it's not circular. And the last thing I think is it doesn't diverge, right? It's like very focused. There is like a clear starting point and a clear end point, right? >> Yeah. >> Okay. I think that's really important. So we said okay understanding what goes looks like being able to actually build it bottom up and improve each step of the way understanding the user journey and exactly which point in time does this bring value and also mapping that within this step it's a linear process of progression right >> I think that's about right

>> okay um what else do you think is really important to to think about when you think about a co-pilot like this >> I think the scoping is a huge part of it >> um like like what we talked about earlier with like the the just trying to understand how much you're trying to bite off cuz you know I I I was teaching a course about both of these interviews but breaking them down into separate ones. We talked about that that's that was an important thing. Um, and then I would say like, you know, interviews kind of were a nice use case to get started with something like this because they have clear evaluation rubrics that, you know, you could help someone understand >> what good looks like or what weak looks like, but a lot of life doesn't have evaluation rubrics, >> right? So I think you know I I think um like great management in general like being clear about expectations and being clear with what looks what good looks like and what does what good does not look like or what bad looks like and being able to articulate that in words um I think will be very important too. um like you you can't I don't think you can get anything out of this kind of product unless you also take the time to kind of create some quantifiable way for it to evaluate its own performance because that's the other thing I would do at the end of the uh the co-pilot will also evaluate its own performance at the end of the interviews. >> Okay. So I think evaluation is another key principle is like the fact that there is some sort of feedback loop between iterations and I think that's critical, right? It's like you don't just do it once. As we said, if it's once, don't build the co-pilot. But you have to also when you finish that linear progression to have clear feedback loop into the entire process. >> Yes. Um and then and and the the other reason that helps is because I knew there was actually two kind of workflows I wanted to bite off. One was how how do how do I work through a question I've never seen before? Like um >> so that was like the the main one, but then also I just filled out this template. Like what can you give me feedback on my filled out template? So to serve the second use case, I had to build the evaluation rubric. And then I realized that if I could solve the second use case with the evaluation rubric that the natural performance I got for the first case was got significantly better because I had codified what the evaluation rubric looked like. >> So um yeah I so my learning from that is you don't necessarily need to be building a product that has like a co-pilot that can evaluate some filled out template or something. But my learning from that was codifying the rubric for evaluation is something that would strengthen the main use case. >> Mhm. >> Mhm. >> The main workflow. >> So Ben, we're about to wrap up and I would like to ask you one final question is like what would be like a key insight you want our listeners to stay with after hearing our uh conversation today?

I think that people people a lot of people dramatically under undervalue how unique their knowledge is about about specific things that they might be the experts on. And I think that in this new world we're going into, >> there's just like a lot of slop. Um, and there's a lot of like I think it's getting harder and harder to know what content to trust and what content is reliable. >> And so my hope is that by talking more about my co-pilot, I don't want people just being like, "Oh, that's so cool." Like I I mean, I think it's cool, but that's not the point of this. The point of this is I want people to be like, "There's nothing particularly special about me." Um, if I can do this, you can do it, too. And but I can't do the heavy lifting for for you as as a listener, as an audience member for what that thing should be and how you do it. That's like the hard part is obviously you could lead a horse to water, but you know, you can't make it drink. Um, and I so I but I do want people to kind of potentially consider building a new type of product that they hadn't considered before. And just because you could vibe code software products now doesn't mean that that's like the best way to maybe package and monetize your knowledge. Maybe there's like a faster path to getting people to what they need. And then as a consumer of your knowledge, like if I'm looking for expertise on, you know, how to improve my website or how to improve conversion in my funnel or how to better how to like these guides that we're doing with insider loops, if someone has like spikes and they're like the top 0.001% when it comes to like information architecture and like guides and like written guides and they could I could like buy something from them that just like allowed us to level up the level up our guides. Like there's so many things areas in my life and I'm sure for you too where you're looking for like what would the best person in the world do at this point with this thing and I just don't think there's like a place to go and discover a lot of that right now and I I would certainly be a buyer for very specific things that fit into very specific workflows for very specific expertise. So that's kind of I guess like maybe we're moving into a world of um kind of like micro micro expertise insertion points and that would be that would be really cool if someone chose to build something out of this. >> Amazing to me is like when we think about whether I want to scale myself and personalize the answers that's a good starting point but the main takeaway from my perspective is resist the urge of building this top down. So don't go and create tons of instructions but build this bottom up. start with the baseline and then add gradually more instructions and make sure that you're controlling the the output. And I think that's key to getting a result that would be gold standard and you can put your name on it. So Ben, thank you so much for uh this conversation. It was really really interesting. >> Yeah, thanks for having me. This was fun. >> And for everyone listening, thank you for listening. You can follow us on all of the podcast software to make sure that you know when a new episode comes in. And again, thank you so much, Ben, for your time. And thank you everyone for listening.