Searching...
16 results for “web scraping”
Concept3 mentions across episodes
Web
...web scraping product, if you need to do occasional web scraping or you have to do web scraping that works every single time, you wanna use browser based. But if you're building web scraping workflows, what you should do is have a waterfall. You shoul
43:34 / 44:36
from transcript43:34 – 44:36
...scraping, which I imagine would be your the bulk of your your workload right now. Right? No. Not at all. I'd say, actually, like, the majority is browser automation. We're Okay. We're kind of expensive for web scraping. Like, I think that if you're b
42:45 / 43:56
from transcript42:45 – 43:56
...that are scraping their own web data for AI training, and then a lot of the other sites they work with are opt in. So companies that want to have their data scraped to help train AI, they honor the robot dot tech settings. They only scrape websites.
32:15 / 33:22
from transcript32:15 – 33:22
...web scraping. Maybe just touch on that. Like, I I guess maybe people, they wanna search and then they wanna scrape. Right? So is that kind of the use case that people have? Yeah. A lot of our customers, they don't just want because they're building A
24:47 / 25:52
from transcript24:47 – 25:52
...and scraping really matters to people. Do you have a perfect scraper? Not yet. Okay. The web is increasingly closing to the bots and the scrapers. Twitter, Reddit, Quora, Stack Overflow. I don't know what else. How are you dealing with that? How are
26:16 / 27:20
from transcript26:16 – 27:20
...scraping, Jason. I didn't actually test that one in particular, but I'm gonna presume that, like, everything else is gonna get shut down. Your point though about Reddit having an API and offering that to people is really interesting because one thing
22:28 / 23:39
from transcript22:28 – 23:39
lacking in data, and so you just try to approximate humans. I I don't know if you guys have seen this. In related news, OpenAI and, Perplexity are going after the browser. Perplexity launched Comet for their $200 a month tier. I actually downloaded i
46:24 / 47:38
from transcript46:24 – 47:38
“This simple tool makes scraping any website for AI prompts effortless”
...website, it was somebody's sub stack. It was Alex's sub stack for cautious optimism. He you then go to it. You say just download the whole McGillic. It won't take the whole website. It'll just take a
47:31 / 48:16
tactical advice47:31 – 48:16
...web and allowing people to do the right thing. And I think the lawsuits are gonna keep piling up. I know that people use RAG and all these things. They they search websites.
22:14 / 23:30
from transcript22:14 – 23:30
...these websites will also block you if you do unnatural behaviors. So people for many time for for all time have been creating local index like Google does. And so they'll just have their browser open a 100 LinkedIn pages and start scraping. And there
1:11:06 / 1:12:15
from transcript1:11:06 – 1:12:15
the top 20 links on this page and summarize them. And it's going up and down the Drudge Report, opening each page, CNBC page, you know, going to each page, summarizing the page, going back, and then loading the next page. That's really interesting if
31:30 / 32:36
from transcript31:30 – 32:36
...they I'm scraping the website. I'm opening the browser windows. So since it's my browser and my IP address and the MAC address of, you know, my computer, it's not their scrapers going and doing things in the world. I think that's like a backdoor hack
12:00 / 13:04
from transcript12:00 – 13:04
and it's actually doing that work. And you can see the steps it's using. And then I can actually open that browser window and watch it do that. This is just a screenshot of it. And it will open multiple of these. So you could I was doing a search the
47:11 / 48:19
from transcript47:11 – 48:19
...web search API like Brave connected, plus they have WebFet, they can go get pages and crawl entire sites. Like, Jason, like, the other thing I wanted to show you folks, this was number three. I have three. Right? Yeah. Is basically how I, I think, Al
47:59 / 49:03
from transcript47:59 – 49:03
...websites to not only block AI scraping and AI pings, but also to set a price for scraping their content. This matters quite a lot in two contexts. The first one is training. We all know that AI models love to go out into the Internet, collect a lot o
3:26 / 5:36
from transcript3:26 – 5:36
Have a podcast?
Run your podcast through Clypt
Get ranked clips, hooks, and ready-to-post copy from your own episodes. Free to try.