home ¦ Archives ¦ Atom ¦ RSS

250K+ More Tweets

Actually, I’ve got over 335K tweets and counting captured on this collection. There were two keys to making progress over my last effort. First, I moved my collector to a personal Linode virtual private server (VPS). As opposed to my ancient home Linux workstation, the VPS can stay up and on the network for days and weeks at a time. No outages due to brief power shutdowns or arbitrary Verizon glitches. Second, my script actually recovered from a network hiccup that killed its child curl process. Picked up the right error code, forked off a new curl, and kept on trucking.

The back of the envelope rate of collection is about 60K+ tweets per 24 hours. At this rate, it’ll take me about another week and a half to hit the million mark. I’m not holding my breath, but glad to see a significant advance over the last milestone.

P.S. After a few months, can thoroughly recommend Linode’s services.

© 2008-2024 C. Ross Jam. Built using Pelican. Theme based upon Giulio Fidente’s original svbhack, and slightly modified by crossjam.