Wide Finder in Clojure

Using Clojure to implement the "Wide Finder", parallel code for grepping log files:

http://technomancy.us/130#c

See the comments for the pmap version:
(ns my-wide-finder
"A basic map/reduce approach to the wide finder using agents.
Optimized for being idiomatic and readable rather than speed.
NOTE: Originally from:
http://technomancy.us/130
but updated to use pmap."
(:use [clojure.contrib.duck-streams :only [reader]]))

(def re #"GET /(\d+) ")

(defn count-line
"Increment the relevant entry in the counts map."
[line]
(if-let [[_ hit] (re-find re line)]
{hit 1}
{}))

(defn my-find-widely
"Return a map of pages to hit counts in filename."
[filename]
(apply merge-with +
(pmap count-line (line-seq (reader filename)))))

http://www.tbray.org/ongoing/When/200x/2009/11/18/Clojure-Parallel-I-O
> Conclusion first: It turns out that Clojure’s concurrency primitives > allow you, with a very moderate amount of uncomplicated code, to > take advantage of parallel hardware and outperform really fast > software when it doesn’t take such advantage.

Posted via email from miner49r