In a recent project, I wanted to do text searches over a large unstructured dataset (100 GB) in memory and I was able to do it in Spark once I provisioned a machine with enough memory. I was able to do it quickly and efficiently, but I was bugged that I couldn't compress the data and had to spin up a master with that much memory.
Read MoreOn my last trip to the farmers market, I spent $20 on a few peaches and a personal watermelon. The fruit was untouchable, but didn’t get me through the weekend (It was that good). I felt good about my decision to support local business and to buy fruit that was in season, but I couldn’t help but feel elite. The atmosphere makes me feel like I’m not only supporting local companies but that I’m better than everyone who isn’t.
Read MoreI bought an Apple Watch Series 2 for Christmas. Most of the motivation for getting it was the struggles I had syncing my workout data. I was using a Garmin Vivosmart watch plus a heart rate monitor to get workout data.
Read MoreDespite having an SEO hostile name, h2o.ai is a pretty cool company. They have developed a great open source plug-and-play data science platform in h2o. They some other projects that are noteworthy and of course Sparkling Water, the subject of this post. Sparkling Water is essentially the h2o APIs on top of Spark, allowing the power of h20 to take advantage of Sparks distributed computing model. That being said, is it worth it to load another dependency when Sparks MLLib is adequate for most machine learning needs? I went through this exercise a few weeks ago and this post is mostly my notes with some added illustration and some code.
Read MoreA few weeks ago I listened to a talk that really got me curious about OCaml. OCaml is a programming language I saw and kind of shrugged off many times in the past. Not because there is anything wrong with the language, just because it doesn’t (at least not overtly) play in any of the domains I play in.
Read More
