Aqui está um pequeno exemplo de como processar big data (com uma pequena amostra) de Clojure usando Apache Spark e a biblioteca Sparkling :
(do
(require '[sparkling.conf :as conf])
(require '[sparkling.core :as spark])
(spark/with-context ; this creates a spark context from the given config
sc
(-> (conf/spark-conf)
(conf/app-name "sparkling-test")
(conf/master "local"))
(let [lines-rdd
;; here we provide data from a clojure collection.
;; You could also read from a text file, or avro file.
;; You could even approach a JDBC datasource
(spark/into-rdd sc ["This is a first line"
"Testing spark"
"and sparkling"
"Happy hacking!"])]
(spark/collect ; get every element from the filtered RDD
(spark/filter ; filter elements in the given RDD (lines-rdd)
#(.contains % "spark") ; a pure clojure function as filter predicate
lines-rdd)))))