Friday, November 20, 2009

Wide Finder in Clojure

Using Clojure to implement the "Wide Finder", parallel code for grepping log files:

See the comments for the pmap version:
(ns my-wide-finder "A basic map/reduce approach to the wide finder using agents. Optimized for being idiomatic and readable rather than speed. NOTE: Originally from: but updated to use pmap." (:use [ :only [reader]])) (def re #"GET /(\d+) ") (defn count-line "Increment the relevant entry in the counts map." [line] (if-let [[_ hit] (re-find re line)] {hit 1} {})) (defn my-find-widely "Return a map of pages to hit counts in filename." [filename] (apply merge-with + (pmap count-line (line-seq (reader filename)))))

Conclusion first: It turns out that Clojure’s concurrency primitives allow you, with a very moderate amount of uncomplicated code, to take advantage of parallel hardware and outperform really fast software when it doesn’t take such advantage.

Posted via email from miner49r

No comments: