Ep #156: When Vibe Coding Breaks, The CMO Pays For It

Transcript

If you’re a chief marketing officer and you wanna become a fractional chief marketing officer, or you are a fractional chief marketing officer and you’re trying to figure out, how do I work with these developers that are vibe coding, or how do I work with my team who’s vibe coding? I wanna show you something that I experienced firsthand, the perils and the worry that I have around it, and how to think about it as a CMO so that you can get faster output for your clients, better outcomes for your clients, and l- less headache.

And let me tell you, if you get this wrong, you’re the one holding the bag. Why? Because a developer can break something, and if they have a bad customer experience, who has to go save the day with marketing-related messaging to save the client’s relationship? You do. So you have to fundamentally understand this vibe coding idea and know when a developer is doing something wrong.

You have to know the right questions to ask. That’s what we’re talking about today

 Hey, welcome back. I’m Casey Stanton, and I wanna show you first on Anthropic’s website, you can see here on my screen if you’re watching this. If not, just trust me. What we’re seeing here is the, it’s a blog post that Anthropic released when they came out with Fable 5 and Mythos 5, and it’s a table, and on this table they’re saying that they’re evaluating Claude Fable 5 against different models, and those models are Opus 4.8.

At the time of recording, Opus 5 is out but also GPT 5.5 and Gemini 3.1 Pro. I just want you to see some values in here. So the first thing that we see at the very top here is that the agentic coding, SWE benchmark, so that’s Software Engineer Benchmark. They’re saying that Fable is touching the 80 percentile for achieving the SWE Bench Pro like outcome.

I read that as coding is really good on Fable 5. Before that Claude Opus 4.8, it’s at, call it 70%, ChatGPT is at 58%, Gemini is at 54%. So agentic coding on Fable is at 80% of reaching a benchmark here for the Software Engineer Bench Pro. That seems really notable to me. And if we look down, agentic coding, so the Frontier Code Diamond, it’s saying that it’s at 29.3%, so that doesn’t seem very high.

But it– as we go down, one of the things that really surprises me here is computer use, huge. Computer use is coming in at 85%. If you don’t know computer use, it is the computer… or the AI’s ability to move your mouse and keyboards and infer what’s on a screen and click stuff. That’s a big deal that we’re reaching this level of proficiency with that.

Biology, it’s 46%. 

The one that is surprising to me am- a- among them all, though, is legal. Legal is down at 13%. Why am I telling you this stuff? This is a bunch of numbers. Here’s why it matters. First, if you’re using an AI tool to try to review legal agreements instead of hiring a lawyer, I’m gonna tell you that’s probably the dumbest thing that you could do right now.

Just have a good lawyer on file that you can call, that you can text. I’ve got a lawyer, I have a question, I send him a message. He bills me for his time, but I am so fast and clear with my questions, and he saves me a ton of time and money. You want a lawyer. You do not want legal agreements read and inferred by these AI tools.

They’re not good enough yet, and I think the risk is too significant. So you want a who there. But when it comes to coding, we see that we’re reaching this 80 percentile for agentic coding on this benchmark. And why I think that is a big deal is because you now have the opportunity or the ability to get coding done faster.

Maybe you do it yourself because you like the stuff and you’re a glutton for punishment, and that’s okay, but I don’t really wanna talk about you doing the coding. I want you to be able to have the taste and understanding of how coding should work. So I’m just gonna give you a couple examples here

All right, I’m gonna grab my iPad, and I have a worksheet here with a triple Venn diagram in it. And I want you to just visualize a triple Venn diagram, right? It’s a three kind of in a triangle shape. And the center where they all connect is an empty space. And what do all companies want? It’s very simple.

They all want the same thing, which is predictable growth predictable growth is predicated on three major things. The first one is just direction. The company needs to be going in the right direction. Go in the wrong direction, they’re going to fail

The next thing that they need is they need brainpower. They need smart people leading. So we wanna put in this second place where these two circles intersect is brainpower

And the third place that they intersect, we want here to be our strengths of deployment, of actual execution

So now, if you’re following along in your mind or if you’re watching this video live on YouTube, you’ll see predictable growth is in the center, then we have direction, execution, and brainpower as the three things where these circles overlap. Okay, in order to get that though, a company needs first marketing strategy.

So that is our big first circle is strategy

The bottom left circle, that’s gonna be for talent and labor

And then to the right, the last thing that we need is leadership

These are the three core fundamental rocks of a sustainable, long-lasting, strong, smart, forward-moving, profitable marketing department that generates predictable growth for their clients. And you know this, right? And as a CMO, your role is to sit squarely on that marketing strategy and marketing leadership side.

And if you had to choose just one of those, I would choose leadership over marketing strategy. Why? Because I can pull strategy from other people, other places, but I want to be the leader. Now, why do I bring all of this up? One of the things that’s gonna happen to you at some point is you’re going to be working with a company, and they’re going to want to leverage AI more.

And how do we leverage AI? There’s the traditional way that people have been using it, which is write this email, write this blog post, remove the em dashes, add a typo, make it look human, write it to our voice, whatever. And that’s okay, right? We’re taking the jobs from the content writers and the poor copywriters.

That’s happening. Okay. But the thing that gets me excited is this agentic AI. So what is agentic AI? And as we look back at our triple Venn diagram here In talent, one of the things that we have for talent is we have AI versus people

So think of the outside of the talent circle. One of the three components of talent is in fact AI versus human. Sometimes it’s AI, sometimes it’s human. We won’t go into the full model, but we have a bunch of other pieces of this that I go into deeper inside of the CMOx Accelerator, where we have nine components that companies are looking for.

But today we’re just gonna focus just on this AI versus human component So you start working with a company and they want you to leverage AI more, and you know that you can do the basic stuff of writing emails and social media posts and that kind of thing. But where things get really interesting is when we start writing software that gets things to happen for us.

So I’m gonna give you a for instance. So I work with a company and they have one of the most awful CRMs that I’ve ever used. And the developers of the CRMs just aren’t doing a good job keeping that CRM current. They have an API that’s really weak. They have really weird rules inside of the system where information kinda gets gated.

And it’s a thing where even if you’re not a technical marketer, you still know it when you get there. You’re like, “I wanna pull this report.” And then you try and you talk to the tech support and they’re like, “You can’t.” There’s a reason that you can’t and that’s frustrating. So let me tell you what I’ve been able to do pretty easily.

I’ve asked Claude Code, I use Claude Code. You could use Cowork, you could use the ChatGPT suites, you could use whatever you want. I’m not here to say that Claude Code is better than anything else. It’s just the one that I use and I will continue to use it just because I know how to use it and it’s easier for me.

Okay? So I use Claude Code and I say to it, “Open up a Chromium window and log into the service,” and then I’ll hop on the computer and type in my username and password. And on my computer, it will go through page after page in the CRM and actually pull down data for me. Think about how incredibly useful that is.

Like I’m pulling reports that aren’t pullable. I’m doing more than just like a VLOOKUP, right? Two separate data sets and matching them by an email address or a phone number. I’m actually having an agent sit at my computer and pull the report for me. And this isn’t like having an intern. This is like having a really smart, capable person like me on my best day, fully caffeinated, without my kids running around, being able to pull all this data and keep it in my mind and pull something that’s actually useful and valuable Having an agent that can do that for me means that I can move faster.

And the layer of getting into that kind of space of being able just to ask this agent to do something really isn’t that difficult, and I think that’s very exciting. But there’s a line between what we’re going to do as CMO types, which is pull the data that we need, get the information that we need to be able to make good distinctions or know what the right next step is for our clients.

But then there’s also, like, all of the back-end data stuff that happens where leads get added, let’s say, to the CRM, and that is something that I would want a technologist to do. I’m never gonna take that responsibility on. I’m not a CTO. That means someone is gonna step in the way of me and this problem and solve it for me.

And more importantly, they’re gonna solve that problem and they’re gonna maintain the solution. This is one of the first things that you have to write down, which is if you’re solving a problem with AI and it uses custom software, custom-developed code, you must have someone maintain that code. And if someone fails to maintain it, it will inevitably break.

Something will happen and it will break. And what I wanna talk about today is knowing or estimating what could break. So I’m gonna give you some use cases here on what could break This is the kind of thing that you need to be able to sit back and think about. And I think a long time ago when the pandemic started and all of these lockdowns happened, and someone wrote on Twitter, they said, “What do you think the second-order consequences of the pandemic is?”

Man, I couldn’t think of them. Could you? In retrospect, can you see some of those second-order consequences? We saw some issues which was like alcohol sales went down and distilleries converted their alcohol equipment into making hand sanitizer, or companies that made neckties and T-shirts started making N95 masks.

Very interesting, like the reinvention of markets because demand had shifted. I want you to be able to think about the second-order consequences of a developer vibe coding. First-order consequences are we got it, it’s fast, it seems to work, we’re all happy, we exceeded the deadline, oh my God, we used this cool technology, everything’s great.

That’s your first-order consequence. Your second-order consequence is it’s gonna break, and you’re gonna be the person who has to catch it when it falls. Not technically, but you have to maintain the company’s image because you’re the marketing person. So here’s how things could fall, f- fail over time.

If you’re doing something on a local computer and it runs a regular job, like a batch job, or we might call it a cron job, after the name Chronos, which means time. So like at a certain interval, every day at 6:00 AM, run this r- thing that goes out, fetches data, pulls it in, sticks it in a Google sheet, and then puts it in Slack.

That’s great until the computer’s offline. So if you have a local computer that runs it, like your computer, great to have but you want a server, which is someone else’s computer that’s on all the time, where there’s people that keep it on. Now, the downside of a server is that they can be expensive.

The positive of a server is you can have a power outage and things can work. I was doing a project where something broke and I put together a short-term vibe-coded fix on a Thursday, and I knew on Monday I’d have the developer back from a vacation and they’d be able to fix it. So I was like, “I vibe-coded a solution.

It’s good enough for now.” It was good enough until literally my power went out for 12 hours. So the solution that I had created wasn’t actually a robust, lasting solution. It was fine. It was certainly better than it would’ve been without me doing that, and at that point, no one else could have done it but me.

That doesn’t mean I wanna be the person that maintains that stuff long term. I was fine to maintain it for something like, 100 hours, but then after that, I wanna be absolved of all responsibility. I could do that because I’m technical and I understand this stuff, but if you’re the kind of marketer who isn’t technical, don’t even try to do things that are mission critical.

Pull someone else in who’s an expert that can do that. Another problem that can happen. So the first one is the hardware that you’re on can turn off, restart, power outage, whatever. Another one is the endpoint that you’re reaching can move. So let’s think of this. If you have keys to a door and you instruct your AI to write some code that take your keys, stick it in a door, open it up, and grab the thing on shelf number two, and then close the door and leave, everything’s fine, right?

That’s gonna work forever until the door moves. What if the door moves three feet to the right? Is that key gonna work? No, ’cause there’s no keyhole for it to go into. So you need to have a– you need to have to consider a process that you’re monitoring those things. And this is an issue with underdeveloped software that you’re vibe coding into.

So I wouldn’t think that something like HubSpot is gonna have this issue. HubSpot has a ton of senior developers that maintain that code base. Full-time developers, yeah, they’re using AI, but they’re maintaining the code base and they’re trying their best not to break stuff on your day-to-day basis. On the other side, you could work with a much smaller startup that has some software, and maybe they’ve vibe coded it, and maybe they broke something, right?

They just moved that door. So now your software, when your custom-coded software tries to reach that door, it’s not there, then what happens? I want you to think about the “then what happens” question because it’s critical to stopping you from falling into a pattern of bombarding an API and getting locked out of creating duplicate records.

So this happened on a project that I was on where someone vibe coded and that endpoint moved, and it moved for two or three days, and we still don’t know why. But like the data in the server logs show that endpoint moved. And the code that was written fine, like it was effective code, but it didn’t back down.

So here’s what happened. It went to go give that lead to the CRM. Typically, the CRM turns around and says, “Thank you for the lead. Here’s their lead ID.” But for some reason, it stopped responding positively. Unbeknownst to the code, that lead was still getting added to the CRM. So the pressure here was the lead was added to the CRM, the software that this developer wrote didn’t know that, and the lead kept getting re-added, and he had no backdown.

So he tried until the API l- uh, rate limited him and said, “Chill out for a while. Take a timeout.” And then when that rate limit expired, he was back at trying again. The result of this was consequential. It meant that leads were re-added to a nurture sequence, and the first step of that nurture sequence was to send a text message.

So now these leads, which opted in, are getting bombarded with text messages all hours of the day, all hours of the night. Painful, brutal. We didn’t notice it until the next workday. This started happening over the weekend. It was a brutal thing to find out. And at the end of the day, the problem was that the endpoint of the API had moved, but that was matched with code that didn’t back off.

Now, those two things are separate companies to me and what I was doing and the client, right? These are, like, hired vendors, so the responsibility is theirs. But I thought to myself, “What did I learn from this?” And I learned that I need to be asking these questions around what happens when you fail?

How does your code back off? How does it try again? Who does it send notice to if it stops working? That’s the second-order consequence you have to think about. So it’s not just what if things break, it’s do they break elegantly? There’s a comic that I like to pass around inside of boardroom at CMX, and it says “A QA engineer walks into a bar and orders a beer.

Orders 999,999 beers. Orders a glass of water. Orders negative two beers. Orders blah, blah, blah,” right? Everything’s fine. Then he asked where the bathroom was, and the whole building caught on fire and burned down. That’s the thing you have to think about on this QA side of stuff.

I gotta be honest with you, it kinda sucks that we can’t just be marketers these days, and we have to stand in this other seat of technology. But if you’re at a company that doesn’t have a CTO, and it’s like a, let’s say it’s a service-based company that is focusing exclusively on serving, and you’re brought in to help grow the business, maybe there’s a sales team, but there’s no tech team.

There might be a website team, but that’s not the same thing. We’re talking about true tech. You start taking on responsibility inadvertently because you’re the only executive in the seat who’s abreast to what’s happening with the technology. So you have to think through these questions, so much so that you probably should just ask Claude or ChatGPT or whatever you use and say, “What questions should I ask the developer for this?

How should I think about this so that we don’t get ourselves in trouble later with rate limiting or adding too many records or breaking something or exposing credentials?” The risk here is significant. They say that something like 50% of the internet runs on WordPress, and for the longest time I’ve used WordPress.

I still use WordPress for some of my sites. WordPress is like a classic content management system, a CMS, like a website builder. And it’s great. The downside is because 50% of the internet is on WordPress, if someone was to find an exploit and they could hack WordPress WordPress sites, maybe a plugin or some file in WordPress or something, then they could get into half of the websites on the internet.

So the vector for WordPress is pretty significant. But the opposite of that is true, which is there’s a lot of developers that support the code base and they make sure that there aren’t issues, and if there are, then there’s critical updates that get passed out and things get updated. So that’s like a mental model for you, which is like WordPress is great because it’s– while it gets attacked a lot, it’s pretty well fortified ’cause there’s people who are paid to protect it.

But think about the other side. You could just vibe code a website now faster than you could build a WordPress website. If you’re c- familiar with Elementor or Divi or one of these other website builders, you could build a website in a day, and maybe there’s some AI tools inside of those builders that makes it a little bit easier.

But there’s nothing as easy as going to Claude, having it generate an HTML website for you, and then pushing that out to Cloudflare Pages or pushing it out to Vercel or Lovable or whatever and getting the page live immediately. You can get there faster than ever before, but you’re doing it with the risk of slop.

Like the way that WordPress works, where there’s a config file and inside that config file has the the username and password and IP address for the database and what the chmod is of that so that the, it’s a protected file so that people on the internet can’t read it. Like that’s really critical stuff, and those things can just get missed by the AI tools, especially if you lack the taste and discernment and experience to know that it’s a potential risk.

So I’m here to tell you uncomfortably that it is slowly also becoming a bit of your responsibility to know what could happen, to see how these AI tools could hurt things

And I’ll tell you, I have never sent an email from my email account from an AI tool. Like once I came back from some travel and I had a bunch of emails and I was just like, “Let me try this Claude thing.” And it did an okay job of like telling me what emails I could delete and summarizing. But honestly, I never want to give up my inbox to an AI.

I want to be the guy who emails from my inbox. That’s critical to me. I had someone email me and it was like an AI email. You could just smell it and just like trust just fell for me, and I’m sure it would for you too. So there are some things that I won’t use AI for. I certainly won’t use it to write for me.

I won’t use it to record these videos for me. But I will use it to build a worksheet or I will use it to vibe code a front-end thing that pulls down from a website data for me. I find now that there’s a ease of using AI vibe-coded tools to interact on my websites, on the CRMs and whatever, or maybe tying to the APIs, but interact there more than I want to develop custom code that I host on a server.

I’d rather run things locally and that to me feels like a line of safety. There’s a tool that you could look into as well if you want to keep stuff really safe, which is Tailscale. That’s a great VPN. It’s free for all of your devices. I use it. I think it’s awesome. Tailscale will let you connect your phone to your computer, to your spouse’s phone.

You guys can be on your own tailnet, which is like an encrypted pipe between all the devices so you can share files and access, your vibe-coded solutions without them being exposed to the rest of the world. The other way that we would do that was we’d have to put it online somewhere.

We’d have to host it at home and open up a port, which is super scary. Or the other way that we’d do it is we’d put it on some website and hope that the credentialing process limited the bad actors from getting into it. I take one level away, like I, I step back from that and I say, “How do I simplify this so it’s best for me?”

Which is Tailscale, locally hosted stuff and limit exposure to the internet. And ask the AI tools “What am I missing on this? What are the potential risks? How should we think about the downsides of this? What could potentially go wrong? How do we make sure we don’t lose as a result of this time savings?”

Like these are the questions that you’re forced to ask as a CMO Too many people are just too busy getting the job done and saying, “Hey boss, I did the job. I’m gonna cut out a little bit early here ’cause I used the AI tool.” You’ve gotta be the person who says let’s figure out if what you’re doing actually is gonna result in a beneficial outcome long term, or if you just solved a short-term problem that’s gonna turn into a big risk exposure for us because you did something stupid.”

These things are critical for you to understand, and that’s actually what we talk about a lot inside of the CMO X Accelerator, is what is working right now, how to use the tools. We should do like a play session once a month with boardroom members where we dive into this stuff and really see what’s working today and how it can work for your clients.

If you wanna learn if I can help you, if you want my personal help building your fractional CMO practice to help you win clients and serve them and really understand how to use AI, like at the cutting edge of what’s possible today, I invite you just to book in a call. Just go to cmox.co/show and book in a call with my team, and we’ll see if you’ve got what it takes for us to help you build a sustainable fractional CMO practice.

All right, I’ll see you later.