There’s a significant difference in the purpose of the scraping.
Google et al. run crawlers primarily to populate their search engines. This is a net positive for those whose sites get scraped, because when they appear in a search engine they get more traffic, more page views, more ad revenue. People view content directly from those who created it, meaning those creators (regardless of whoever they are) get full credit. Yes, Google makes money too, but site owners are not left in the cold.
ChatGPT and other LLM’s works by combing its huge database of known content its “learned” to cook up an answer through fast math magic. Content it scrapes to populate this database can be regurgitated at any time, only now its been completely processed and obfuscated to an insane degree. Any attribution of content is completely stripped in the final product, even if it ends up being a word-for-word reproduction. Everything OpenAI charges for its LLM goes directly to OpenAI, and those who have created content to train it will never even know it was used without their consent.
Essentially, LLM’s operate like a huge middle school plagiarism machine shitting all over any concept of copyright, only now they’re making billions off said plagiarism with no plans to stop. It’s a huge ethical conundrum and one I heavily disagree with.
No the real reason for the API changes was to shut down apps; and the reason for that is because the apps gave users too much freedom to not be a perfectly packaged product for the real customer: advertisers, and others payed promotion.
How do we know this? Simple. a) Shutting down APIs does nothing to prevent dedicated content scrappers, b) it would have been totaly possible to lock down APIs and negotiate fair deals with app developers, to continue third party apps while having the same rate limiting on scrappers as what we have now, and c) this all coincided with some bigger picture business model changes at Reddit, including Reddit For Business, Reddit’s IPO, and the reduced VC funding in the tech industry at large.
To blame that saga on AI scrappers really obfuscates the fact that Redditors are just cattle who’s eyeballs are to be packaged and shipped to the real paying customer.
Google et al. run crawlers primarily to populate their search engines. This is a net positive for those whose sites get scraped, because when they appear in a search engine they get more traffic, more page views, more ad revenue.
This is not necessarily true. Google’s instant answers are designed to use the content from websites to answer searcher’s questions without actually leading them to the website. Whether you’re trying to find the definition for the word, the year a movie came out, or a recipe, Google will take the information they’ve scraped from a website and present it on their page with a link to the website. Their hope is that the information will be useful enough that the searcher never needs to leave the search engine.
This might be useful for searchers, but it doesn’t help the sites much. This is one of the reasons news companies attempted to take action against Google a few years ago. I think a search engine should provide some useful utilities, but not try to replace the sites they’re ostensibly attempting to connect users to. Not all search engines are like this, but Google is.
There’s a significant difference in the purpose of the scraping.
Google et al. run crawlers primarily to populate their search engines. This is a net positive for those whose sites get scraped, because when they appear in a search engine they get more traffic, more page views, more ad revenue. People view content directly from those who created it, meaning those creators (regardless of whoever they are) get full credit. Yes, Google makes money too, but site owners are not left in the cold.
ChatGPT and other LLM’s works by combing its huge database of known content its “learned” to cook up an answer through fast math magic. Content it scrapes to populate this database can be regurgitated at any time, only now its been completely processed and obfuscated to an insane degree. Any attribution of content is completely stripped in the final product, even if it ends up being a word-for-word reproduction. Everything OpenAI charges for its LLM goes directly to OpenAI, and those who have created content to train it will never even know it was used without their consent.
Essentially, LLM’s operate like a huge middle school plagiarism machine shitting all over any concept of copyright, only now they’re making billions off said plagiarism with no plans to stop. It’s a huge ethical conundrum and one I heavily disagree with.
And pretty sure this is the catalyst for reddit’s API changes. Other companies are getting rich off of them and they want a piece of the pie.
No the real reason for the API changes was to shut down apps; and the reason for that is because the apps gave users too much freedom to not be a perfectly packaged product for the real customer: advertisers, and others payed promotion.
How do we know this? Simple. a) Shutting down APIs does nothing to prevent dedicated content scrappers, b) it would have been totaly possible to lock down APIs and negotiate fair deals with app developers, to continue third party apps while having the same rate limiting on scrappers as what we have now, and c) this all coincided with some bigger picture business model changes at Reddit, including Reddit For Business, Reddit’s IPO, and the reduced VC funding in the tech industry at large.
To blame that saga on AI scrappers really obfuscates the fact that Redditors are just cattle who’s eyeballs are to be packaged and shipped to the real paying customer.
In light of your statement / argument, I’ll reframe similar responses in the future. What you say makes sense. Thanks.
I think you are both right to be honest. It’s not mutually exclusive.
This is not necessarily true. Google’s instant answers are designed to use the content from websites to answer searcher’s questions without actually leading them to the website. Whether you’re trying to find the definition for the word, the year a movie came out, or a recipe, Google will take the information they’ve scraped from a website and present it on their page with a link to the website. Their hope is that the information will be useful enough that the searcher never needs to leave the search engine.
This might be useful for searchers, but it doesn’t help the sites much. This is one of the reasons news companies attempted to take action against Google a few years ago. I think a search engine should provide some useful utilities, but not try to replace the sites they’re ostensibly attempting to connect users to. Not all search engines are like this, but Google is.