Friday, November 20, 2009

Wide Finder in Clojure

Using Clojure to implement the "Wide Finder", parallel code for grepping log files:

See the comments for the pmap version:
(ns my-wide-finder
"A basic map/reduce approach to the wide finder using agents.
Optimized for being idiomatic and readable rather than speed.
NOTE: Originally from:
but updated to use pmap."
(:use [ :only [reader]]))

(def re #"GET /(\d+) ")

(defn count-line
"Increment the relevant entry in the counts map."
(if-let [[_ hit] (re-find re line)]
{hit 1}

(defn my-find-widely
"Return a map of pages to hit counts in filename."
(apply merge-with +
(pmap count-line (line-seq (reader filename)))))
> Conclusion first: It turns out that Clojure’s concurrency primitives > allow you, with a very moderate amount of uncomplicated code, to > take advantage of parallel hardware and outperform really fast > software when it doesn’t take such advantage.

Posted via email from miner49r

No comments: