Don’t know how I missed this, but O’Reilly and Cloudera combined Hadoop World and Strata. It’s going to be in New York, in October. Perfect topic, timing, and location. We’ll have to see what the registration costs are and of course New York isn’t exactly the cheapest city to visit.

Meanwhile, SciPy 2012 will be in July in Austin. Downsides? Another Python conference and Texas in July. Upsides? Another Python conference and it’s Austin, Texas. With friends in the area, it’s very tempting.

And I’d still like to squeeze in a David Beazley Chicago Python course.

“Having seen the video, I find that I most often refer to the Magit Cheatsheet and the Magit User Manual. If you’re an emacs and git user, you probably owe it to yourself to spend 15-20 minutes watching the video and trying out magit. I think you’ll quickly realize how useful it is.”

That would be me sir. I’ve been using magit regularly but in the most naive of fashions. So I will be following your suggestion posthaste.

1 BILLION TWEETS! (curling lip ala Doctor Evil and chuckling diabolically)

That would actually be something outstanding for a single individual. Consider that to do it in a calendar year, you’d have to average just a little south of 2.75 **million* tweets per day, 114K per hour, 31 per second. I’m not even sure that’s possible with Twitter’s streaming API. And then there’s the bandwidth and storage issues. Not to mention maintaining pretty high annual uptime to stay on target.

Now that would just be the capture. What the heck would you then do with all that data? Imagine the possibilities.

I think a reasonably well funded individual could pull it off with careful exploitation of Amazon Web Services, but it would definitely be non-trivial. I can dream can’t I!

Better. Than. Remembered.

The argument parsing really easy and convenient. I am now salivating about the logging and daemonizing features. pyCLI doesn’t do anything spectacular, but it makes a couple of good to do things, easy to do.

After an epic hunt across the Interwebs, I finally managed to find a solution to enable Thunderbird Spotlight indexing thanks to GetSatisfaction. Suffice it to say it involves gnarly file permission tweaking.

Then I got to thinking, wouldn’t it be great to limit Spotlight searches to just e-mail? Gotta be a way to do that right? Turns out Spotlight supports a number of keywords to match various file types. kind:email gets both Thunderbird and Outlook messages. Handy!

“I don’t believe that’s true though. There is an optimistic way of understanding my first point, and that’s my second point: Even if you aren’t the smartest person around, and your product is kind of ugly and broken, you can still be very successful, if you just build the right product. YouTube and MySpace are both fine examples of this.”“

…“When Google acquired YouTube, many people inside the company were flabbergasted, “But they have no technology!?” They didn’t understand that you only need enough technology to make the product work.”

I’ve been trying to bring a more entrepreneurial mentality to work. Unfortunately, the tendency is towards trying to game the customer, rather than building “products” they want. Now we’re in science and technology applied research for the global security market, so product is somewhat ill-defined. But I know one thing, more slideware doesn’t cut it these days. Unfortunately, here’s the pervasive tendency within my org. 1) Think up some cool idea, 2) put it in PowerPoint, 3) shop it to program managers, 4) PROFIT (or not).

And I should really know what Buchheit is saying, given what I’ve seen recently, but it’s always a struggle. Enough technology, or data, or analysis, can make the product, our intellectual services, work.

I’ve personally seen it rip through approximately 4 Gb of data on a 5 year old server with only 8 Gb of RAM, said data on a suboptimal Linux ext3 filesystem, on top of an untuned kernel, and with no thought given to the IO and HD subsystems. Grand total of 21 minutes.

That is a nice capability to have.

I know Lucene with Solr on top is sort of the default open source choice for full text indexing, but if you’re in the market Sphinx is worth a tire kick.

Hadoop World 2012 in the Fall is fairly attractive.

Hat tip: Jon Zaunich

Rapidly updated text indexing from folks who really have to care about it.