Wide Finder in Clojure

Using Clojure to implement the "Wide Finder", parallel code for grepping log files:

http://technomancy.us/130#c

See the comments for the pmap version:

(ns my-wide-finder "A basic map/reduce approach to the wide finder using agents. Optimized for being idiomatic and readable rather than speed. NOTE: Originally from: http://technomancy.us/130 but updated to use pmap." (:use [clojure.contrib.duck-streams :only [reader]])) (def re #"GET /(\d+) ") (defn count-line "Increment the relevant entry in the counts map." [line] (if-let [[_ hit] (re-find re line)] {hit 1} {})) (defn my-find-widely "Return a map of pages to hit counts in filename." [filename] (apply merge-with + (pmap count-line (line-seq (reader filename)))))

http://www.tbray.org/ongoing/When/200x/2009/11/18/Clojure-Parallel-I-O

Conclusion first: It turns out that Clojure’s concurrency primitives allow you, with a very moderate amount of uncomplicated code, to take advantage of parallel hardware and outperform really fast software when it doesn’t take such advantage.

Posted via email from miner49r