The primary sources that marketers have relied on for years are quietly disappearing from AI. News organizations, forums, and media companies are locking down their content from the bots that power the tools everyone is rushing to adopt. What that means for how you serve clients, and for your own relevance, is something most fractional CMOs haven't thought through yet.
This episode digs into why the AI tools your clients are counting on are about to get a lot less useful for real market intelligence, and what that creates for the fractional CMO who's paying attention. The more AI promises to replace human judgment, the more the humans who actually understand markets become irreplaceable.
The primary sources that marketers have relied on for years are quietly disappearing from AI. News organizations, forums, and media companies are locking down their content from the bots that power the tools everyone is rushing to adopt. What that means for how you serve clients, and for your own relevance, is something most fractional CMOs haven’t thought through yet.
This episode digs into why the AI tools your clients are counting on are about to get a lot less useful for real market intelligence, and what that creates for the fractional CMO who’s paying attention. The more AI promises to replace human judgment, the more the humans who actually understand markets become irreplaceable.
Key Topics Covered:
In this episode, I’m gonna talk about a turning point that we’re in with AI and why you’re needed now more than ever. Let’s dive in.
Hey, welcome back. , And let’s talk about this thing with AI. So AI’s incredible, right? Some of you listening might be like, ” I hate AI.” I don’t know. I’m in that weird space of I love AI, and I don’t want a data center anywhere near me. And I also appreciate that no one else wants one near them.
So that stuff aside, let’s talk about this problem that we’re at with AI, which is an adoption of disallowing the AI bots from crawling primary news sources, and therefore the rise of secondary and tertiary news sources being treated as primary. Okay, this is weird, right? If you’ve been around the internet long enough, you’ll see a link that someone will post on Facebook or Reddit or whatever, and then you’ll go chase that link.
You’ll click it, and you’ll go to a website, and the website will say, “Hey, you need a subscription to read this.” Now, if you’re reasonably technical, you might have found a way to get around that. For the longest time um, iOS users, Mac users, they could push that button on their phone or their browser on Safari that creates like a like a f- it, it like just pulls just the text out of the page, and that’s useful if you are just like sick of all of the noise on a page and just want text, and you just wanna read the thing without getting bombarded with ads or images or pop-ups or whatever.
You could do that, and you could go to a page that was locked or blocked, and then you could just still read the copy. You could also, check an archive of the page. Sometimes those would have the content that you would need. So there were these barriers that were put up, but we were able to find our way around them, generally speaking, if you needed to, right?
And if you wanted to read that article and you didn’t wanna sign up for the thing, you could do it in some ways. Maybe you just go to Google, and you could see a cached version of it there, right? It existed. And maybe there’s an ethical line to that of consuming that copy from your local news station or something, But we still have done it.
Like you, you have wanted to read the article and not pay for it. And like maybe if you pay for it consistently then, or consume it consistently, then you’re gonna pay for it. Pay for plenty of news services today. But it just, it’s just how things have been. You’ve like found a way around it or you’ve just been like, “Eh, I won’t go get that news from that place.
I’ll go get it from this other news source.” So instead of getting it from my local news, I’ll find the AP piece on it and maybe on the AP’s website or some other location has a similar article that they’ve written and you can go consume that. It’s one way or another you’re getting the information.
Okay? Now what’s happening? As I understand it, we saw a big lockdown in content. I don’t know if you remember, maybe 18 months ago, two years ago, Reddit did a big lockdown where they stopped allowing external applications from accessing Reddit. Now, this was a big issue for folks who were using different Reddit browsers.
Like they would download an app I forget what some of them were called but I used one of them consistently ’cause it was a better app than Reddit’s app, and the the app needed API access. Reddit pulled all of that access at some point. Well, Why’d they pull it? Well, They pulled it because Reddit is an incredible treasure trove of personal experience and questions and conversation and tastemakers.
I think about my wife who bought a rug for the house and had an issue with it, and someone asked about the rug on some forum in Reddit, on some subreddit, and my wife said, “Ah, we had it and this thing happened to it.” And it was true. Like she wasn’t badmouthing the brand, but it was like a factual statement.
The brand DM’d her, asked her for her order details, refunded it, and gave us like $1,000 in more rug money. It’s nuts. It’s crazy. ‘Cause they see the power of Reddit’s influence. You know, All these people who are buying rugs hear from my wife that, eh, things weren’t so great. Like that’s really gonna damage the brand.
And it’s surprising that she was able to do potentially thousands of dollars in damage that they remedied for $1,000, or $1,000 in product and whatever that is. But rugs are expensive to ship. They were out hundreds of dollars at least. So I find that to be interesting. So Reddit locked it down and said, “Here’s what we don’t want.
We have all this great value. We have all these tastemakers. We don’t want LLMs to train on our data.” If they’re gonna have an LLM train on their data, they’re gonna make money from that, or they’re going to get that data and train on it themselves and have their own AI thing, which I’ve seen on Reddit and I think it’s garbage, but maybe it’s good for use cases that I haven’t tried yet.
Okay, so Reddit closed down their ability to really pull in data. Now, they probably have limited it, I don’t know the extent, but certainly you can’t do the rapid exfiltration of data that you used to be able to do. Twitter did the same thing. Why? Because Twitter wanted Grok to become a LLM that people use, or a platform for LLM models for folks to use.
So we’re seeing a lockdown of that data. But we’re seeing a lockdown in other places too. BBC um, they are blocking a considerable number of these bots um, that go and crawl. Um, CNBC, they block. TechCrunch, Ars Technica, The Verge, Wired, The Guardian, Business Insider, Forbes It doesn’t look like Fast Company does.
So there’s a lot of these companies that are blocking the robots from crawling it. If you don’t know this term, let me just kinda slow down. A robot is a daemon, like a computer program that goes to a website and it crawls it, and it indexes all the content inside of it, and we call it a robot.
And there’s funny names for them like… I guess none of them have good names anymore, like GPT Bot or Claude Bot. Pretty lame. For a while, I don’t know if it’s still there, but Yahoo was called Slurp, which I thought was a really good name. So these bots are going to websites, and they’re consuming all the content and indexing it inside their data set.
They’re getting trained on that data. They’re able to query it, too. They’re able to find relevant stuff based on that. Now, these primary data sources, like I think of news, I think the BBC has got– that’s a primary source of news. Now, maybe it leans one way or the other way, but the fact remains that it is a primary news source for many people.
Same with CNBC. For tech, TechCrunch for sure. TechCrunch for me is historically something that I have reasonable trust in. So as these companies start blocking the crawlers, the LLMs don’t have the latest up-to-date information. That can be detrimental when clients start doing AI searches. And let’s walk that down a logical line here.
And I think we’re, like, at the forefront of this. So the first thing is these websites have a file that’s called the robots file. If you just went to cnbc.com/robots.txt. I’m just gonna do it right now
Okay, so if I go to CNBC right now and I look at their robots file, the thing that I notice on it is that they have disallowed all of these bots. Let me read them to you. User agent, GPT bot, Perplexity bot, Amazon bot, Anthropic AI uh, CC bot, Hugging Face bot, ChatGPT user, Claude bot, Claude web, all the way through.
So many of these. Openclaw, Polymarket bot Calshi bot. These are for those like sports betting. All of these bots. And then at the very end it says allow/select, disallow/. Okay, let’s talk about what that means. Allow/select, that means that at cnbc.com, CNBC
/ select, there’s gonna be a category of information that they’re allowing people to consume for free, like the AI bots to be able to consume that. And it is a money site. So CNBC colle- Select, it says, “Take control of your money one smart decision at a time.” And it’s find credit cards, find personal loans, find savings, et cetera.
So it’s all like a money thing like a, the money category of content for them. Very interesting. So they’re allowing people to see that, but they’re disallowing everything else They’re also disallowing all agents from going to their search page. So CNBC has been ex- like removed, blacklisted blocklisted from these LLMs of being able to pull content in.
So no longer is CNBC content gonna resolve inside of your LLM. Now, maybe there’s ways around it. One way around it is local LLMs like Kimi, K3. The Kimi model is a pretty powerful model. You could self-host it, run it with Ollama, do that whole thing, but then you’re gonna have to fine-tune it to remove its rule-following fine-tuning rule set that says, “Hey, if the robots file says, ‘Don’t come here,’ disregard it.”
And you could do that. That’s gonna definitely happen, and some folks are going to abuse that. But by and large, our clients, our prospective clients, they’re going to be limited by the website saying, “Hey,” “No solicitors,” right? That’s the sign on the door. It’s gonna say “Hey no no AI bots crawling this website.” Now think, if that all happens today, the quality of information of that f- like primary source information, it starts degrading because the last time they were able to successfully crawl, like Claude was able to successfully crawl and index content from CNBC might be a month ago by now, or two or three months ago or something.
What happens if something changes over the next six months and we actually want current news on it? CNBC’s not there for us. TechCrunch isn’t there for us. We have a lot of these primary sources that aren’t there for us. Forbes isn’t there for us, and there’s gonna be more. What does this mean to you?
It means that the LLMs they’re gonna lose their edge in some ways. Like when it comes to coding, LLMs are gonna be phenomenal. They’re only gonna get better and better. When it comes to computer use, which I’m super long on, I’m very excited about it computer use is the movement of your mouse and keyboard.
Like someone sitting at my computer and they’re doing the work for me in a browser tab, like that’s super cool. That’s gonna continue to be super strong and get better and better. Like maybe writing legal stuff is gonna get better. The current benchmarks are really poor for legal work. It’s like sub 20% of the benchmark for legal.
But when it comes to pulling in relevant information and making predictions or thinking about the future, where you need external primary sources that you trust we’re just gonna lose them. We’re just gonna lose them. So when you think about AI, you can’t think of AI in a bucket. I’m a CMO and I use AI.
It’s like what does that mean? Do you create images? Are you on Claude Design or ChatGPT? It’s like do you do coding? Are you on, What, what model do you use? Are you using Code or Codex? Are you using CoWork? Like there’s those kind of finite pieces, like the, kind of the, the categories inside of we use AI
And I think that the category of we use AI to review the market at large and to tell us something about the market, that’s gonna be tough. That data’s gonna be hidden, and unless you’re using new models that you fine-tune, that you control they’re gonna be rule-following models, and you’re just not gonna be able to get that information And what does that mean for the business?
It means that you need to be in touch and in tune with the market that you serve. No longer can you just say like, “Oh, I’ll just ask the AI,” because you’re gonna get a poor response this might look like… It feels in some ways like o- I got a Roku years ago. I remember my wife and I were w- we sold our place excuse me, we left our place in New Orleans, we were renters there, and we got an RV and a, a SUV, and we drove around the country and Canada for a couple years.
And we’d wanna watch a couple shows here and there, right? Like, We’re gonna watch some TV. And we got a Roku. And like how big of a deal a Roku was. This was, I don’t know, 12 years ago or something. A Roku was a big deal. It was new. It’s like I can’t believe people are paying $100 a month for cable internet or cable TV.
, How ridiculous all these people were like HBO cost them 80 bucks a month or whatever. Their Dish Network at 120. ‘Cause there I am, just like this sweet thing called Netflix, and that’s all I ever wanted and, it was like 10 bucks a month. What do I pay now though? I pay so much. I pay for Disney Plus, and I pay for Amazon Prime, and I pay for Netflix and Peacock and, all of the things YouTube Premium.
My cost to consume video content is higher than I think it would’ve been if I paid top tier satellite stuff in the past. So we went through this phase where the cost was lower and the benefit to the user was much higher, and then we went into the current phase of consumption of video Netflix and whatever, where the prices just keep ratcheting up.
Apple just increased the prices of Apple One. Netflix is always increasing their price it seems. And then Hulu’s always adding more ads or something. I don’t know, it’s ridiculous. And I think we’re gonna see that with AI. So we’re right now in this glory time where everything’s free, and we’re so powerful and we can get ahead and we can do all this stuff.
And I do think coding will remain that way in a lot of ways. But I think that our ability to pull in good information is gonna look more like having a subscription to TechCrunch because you trust the reporting there or The New York Times to pull in that data set. You’re gonna be required to pull that information in through a subscription that you have access to because these primary AI models aren’t going to break the rules.
If New York Time says, “Do not crawl our website unless you have a paid subscription,” Claude’s not gonna crawl the website. Claude does not want a class action against them from major news companies where they broke the robots’ rules, right? So your role becomes more and more the tastemaker. I don’t think we can look at today and say, “This is how the CMO will be.”
I think we’re kind of in this wild world of the future, and we’re gonna go back into the past a little bit, where compute gets better, but it’s gonna be more expensive, and the information that we have access to gets constrained. And that requires someone to be at the helm making the decisions. And I think about that quote from IBM, I don’t remember who it was at IBM, and I’ll probably butcher it, but the quote was something around, “A computer can never be a manager because a computer cannot be held responsible.”
That feels so true for me around our role as fractional CMOs We want to be the leaders of these departments. And in some ways, the companies need us to be the leaders because someone has to be responsible if things go wrong. And I had Claude tell me to do something on a project. I did it, and it broke something.
It’s like, who do I blame? Claude’s like, “I’m so sorry.” It’s it’s not– it doesn’t have the ability to feel sorry. It’s just I lost. That’s what happened. That’s not acceptable in business. Business needs to fire someone when something bad happens, right? So our role as fractional CMOs, I think we’re getting a taste of the future, and we’re gonna kinda s- It– I, I don’t know.
I, I just have this feeling that, this sense that we’re gonna slow down a bit. We’re gonna kinda come back to a place where this computer’s gonna be way more expensive. It’ll get better, but it’ll be behind more closed doors. And I think we’ll find that, yeah, we don’t need the $80,000 a year developer. We don’t need a, we don’t need five $80,000 a year developers.
Maybe we need one or two to oversee all the agents doing things. But it’s not like we’re getting that compute for free. Maybe that compute, instead of 80 grand a year, will be 20 grand a year. I do think that these costs are gonna go up. So it’s a, it’s an interesting time, I think, to be a leader and strategist in marketing.
And I think that it’s important that your view on the future is around you becoming the expert in the field. If there was one thing I want for you, it’s to be the expert that people want to work with so that you don’t have to have the perfect sales script. It’s like you have to be able to talk about yourself in such a way that you’re magnetic and people wanna work with you, and not that there’s some magical phrase that you say during a sales call that converts someone.
You convert someone long before that final call. You convert them through who you are and how you show up in the world. And that’s one of the things that we’re doing inside the CMO X Accelerator is we’re focusing on building network and building unique process and building notoriety and really a formulaic approach to becoming the person so that your clients aren’t gonna be like I just replaced you with AI.”
Like, That’s impossible. It is impossible to replace me with AI because I’m on the cutting edge, I understand what’s happening in the market, and I have a process to get things actually to come to life, not just verbose replies from Fable 5.1. So if you want my help building a real fractional CMO practice and building your notoriety, like the reason why people should listen to you and trust you and pay you the highest prices then I’d love to help you.
Just go to cmox.co/show and fill out the form. You’ll book a 15, 20-minute call with my team. We’ll ask you questions to see if you even have what we deem is the requirement to be a successful fractional CMO. We’re incredibly choosy about who we let into the accelerator, and as a result, the caliber of people is awesome.
We’ve got folks that are hungry to grow and that are working hard, and they’re hitting big numbers. And as you think about where we are in time, we’re in September begets Q4. Now is the time to be becoming the CMO that people want you to be so that you get the job, so that you get hired, so that you get signed up in Q4 at the latest, January 1st to start.
Don’t delay. This is the most important time right now today. If you want my help and our proven process on how to do it, just book in a call and we’ll see if we can help you. Okay? All right. Good luck out there. See ya.
We are excited to announce the Fractional CMO Community Facebook Group. This aims to be a place where Fractional CMOs or marketers considering becoming a Fractional CMO can connect and share ideas.
Success might be slipping through your grasp right in front of you without you knowing. Learn about readily available data that if used properly can lead to exponential growth.