been poking data universe for a bit. the pitch is great, decentralized data scraping, big numbers on the dashboard. but every time i actually try to pull something usable the quality is all over the place. half the rows are stale or duped. anyone actually getting clean data out of this or is it just volume for the sake of volume
volume for the sake of volume is also my gym routine and look how thats going lol
nah i think the dupes are kinda the point of a scraping net 😅 you filter on your end 🧹 expecting clean rows out the box feels off 🤷
stale and duped is kind of the whole game with open scraping though, you cant push back on volume and then complain its volume. the dedup is on you at the consumer end, thats how every data network works. it isnt supposed to hand you clean rows
same experience. volume is real, signal is not. you spend more time cleaning than the data is worth
数据量是真的猛啊!!! 我这边拉下来洗一洗能用 别只盯着脏行 整体方向我看好 起飞只是时间问题
what source were you pulling. some are way cleaner than others
disagree, the dupes are a validator scoring problem not a you problem. if the incentive rewarded freshness youd see better rows. the network can fix this it just doesnt
did you try filtering by validator score or just pull raw. that changes the dupe rate a lot for me
agree. volume for the sake of volume. ive pulled it. its junk until you spend a day scrubbing
you discovered that free scraped data is messy. groundbreaking. next youll tell me the ocean is wet. clean it yourself like everyone else
big numbers on a dashboard, low signal in the pull. you have described approximately every data subnet ever shipped
本来跟你一样吐槽数据脏 后来换了个source 加了去重 真香!!! 清洗一下完全能用 别急着否 这个是有东西的
同感哈哈 我拉了一批 一半是重复的 量大但是脏 得自己写脚本清 不然真没法直接用
