Redis & Groovy integration for caching and messaging

Redis & Groovy & Grails

by Ted Naleid
http://naleid.com

Monday, June 20, 2011

“Redis is a collection of data
structures exposed over the
network”

from: http://nosql.mypopescu.com/post/5403851771/what-is-redis


key/value store

like memcached on steroids


Strings, Integers,
Lists, Hashes,
Sets & Sorted Sets
(& commonly expected operations with each data type)


in-memory storage


single-threaded


“Memory is the new Disk,
Disk is the new Tape”
- Jim Gray


Relative Latency
CPU Register - 1x
L2 Cache - 10x
Memory - 100x
Disk - 10,000,000x

analogy from “Redis - Memory as the New Disk” - Tim Lossen &
http://en.wikipedia.org/wiki/Orders_of_magnitude_(speed)

CPU Register
1 yard

photo: http://www.ﬂickr.com/photos/limonada/904754668/


L2 Cache
10 yards

photo: http://www.ﬂickr.com/photos/plentyofants/2749262107


Memory
100 yards

photo: http://www.ﬂickr.com/photos/billmcintyre/264905933


Disk
Minneapolis to New York to Miami
to Seattle
~5600 miles


Simple wire protocol that
matches API


% telnet localhost 6379
Escape character is '^]'.
set foo bar
+OK
get foo
$3
bar
rpush mylist first
:1
rpush mylist second
:2
lrange mylist 0 -1
*2
$5
first
$6
second


clients for every* language

*well not every language, but all the popular/semi-popular ones, you can easily write one if
your language doesn’t have one


No dependencies
(451KB download)


Used in production by high
traffic sites that you’ve used


stackoverflow, craigslist,
github, disqus, digg &
blizzard entertainment


financially supported by
VMware/SpringSource


simple data structures make
redis flexible


write-through caching/
memoization


producer/consumer
message queues


publish/subscribe


atomic sequences/counters


other uses...
distributed locks, tag clouds, session tokens,
auto-complete prefixes, API rate limiting, leaderboards,
capped logs, random set items, A/B testing data storage,
unique per user product pricing/sorting


Redis Commands


Great Online Reference
http://redis.io/commands


Strings
set

Redis REPL Groovy

> set foo bar redis.set("foo", "bar")
foo bar
OK OK


Strings
get

Redis REPL Groovy

foo bar

> get foo redis.get("foo")
"bar" "bar"

bar


Exists
exists (check key existence)

Redis REPL Groovy

foo (nil)

> exists foo redis.exists("foo")
(integer) 0 <= false

0


Integers
incr (increment)

Redis REPL Groovy

foo 1

> incr foo redis.incr("foo")
(integer) 1 <= 1

1


Integers
decr (decrement)

Redis REPL Groovy

foo 0

> decr foo redis.decr("foo")
(integer) 0 <= 0

0


Lists
rpush (right push)

Redis REPL Groovy

> rpush foo baz redis.rpush("foo", "baz")
foo baz
(integer) 1 <= 1


Lists
rpush (right push)

Redis REPL Groovy

> rpush foo qux redis.rpush("foo", "qux")
foo baz qux
(integer) 2 <= 2


Lists
lpush (left push)

Redis REPL Groovy

> lpush foo bar redis.lpush("foo", "bar")
foo bar baz qux
(integer) 3 <= 3


Lists
lrange (slice of list)

Redis REPL Groovy

foo bar baz qux
> lrange foo 0 -1 redis.lrange("foo", 0, -1)
1) "bar" <= [bar, baz, qux]
2) "baz"
3) "qux"
bar baz qux


Lists
lrange (slice of list)

Redis REPL Groovy

foo bar baz qux
> lrange foo 0 1 redis.lrange("foo", 0, 1)
1) "bar" <= [bar, baz]
2) "baz"
bar baz


Lists
lpop (left pop)

Redis REPL Groovy

foo baz qux

> lpop foo redis.lpop("foo")
"bar" <= "bar"

bar


Lists
rpop (right pop)

Redis REPL Groovy

foo baz

> rpop foo redis.rpop("foo")
"qux" <= "qux"

qux


Hashes
hset (set key → value)

Redis REPL Groovy

> hset foo bar baz redis.hset("foo", "bar", "baz")
foo bar baz
(integer) 1 <= 1


Hashes
hset (set key → value)

Redis REPL Groovy

bar baz
> hset foo qux quxx redis.hset("foo", "qux", "quxx")
foo
(integer) 1 <= 1
qux quxx


Hashes
hget (get value for key)

Redis REPL Groovy

bar baz
foo
qux quxx
> hget foo bar redis.hget("foo", "bar")
"baz" <= "baz"

baz


Hashes
hgetall (get all keys/values)

Redis REPL Groovy
bar baz
foo
> hgetall foo qux quxx

1) "bar" redis.hgetAll("foo")
2) "baz" <= [bar:baz, qux:quxx]
3) "qux"
4) "quxx" bar baz

qux quxx


Hashes
hvals (hash values)

Redis REPL Groovy
bar baz
foo
qux quxx

> hvals foo redis.hvals("foo")
1) "baz" <= [baz, quxx]
2) "quxx"
baz quxx


Hashes
hkeys (hash keys)

Redis REPL Groovy
bar baz
foo
qux quxx

> hkeys foo redis.hkeys("foo")
1) "bar" <= [bar, qux]
2) "qux"
bar qux


Sets
sadd (set add)

Redis REPL Groovy

> sadd m1 jan redis.sadd("m1", "jan")
m1 jan
(integer) 1 <= 1


Sets
sadd (set add)

Redis REPL Groovy

feb
> sadd m1 feb redis.sadd("m1", "feb")
m1
(integer) 1 <= 1
jan


Sets
sismember (membership test)

Redis REPL Groovy
feb
m1
jan
> sismember m1 jan redis.sismember("m1", "jan")
(integer) 1 <= true

1


Sets
sismember (membership test)

Redis REPL Groovy
feb
m1
jan
> sismember m1 mar redis.sismember("m1", "mar")
(integer) 0 <= false

0


Sets
smembers (get full set)

Redis REPL Groovy
feb
m1
jan
> smembers m1 redis.smembers("m1")
1) "feb" <= [feb, jan]
2) "jan"
feb

jan


Sets
sinter (set intersection)

Redis REPL Groovy
feb feb
m1 m2
jan mar
> sinter m1 m2 redis.sinter("m1", "m2")
1) "feb" <= ["feb"]

feb


Sets
sdiff (set difference)

Redis REPL Groovy
feb feb
m1 m2
jan mar
> sdiff m1 m2 redis.sdiff("m1", "m2")
1) "jan" <= ["jan"]

jan


Sets
sunion (set union)

Redis REPL Groovy
feb feb
m1 m2
> sunion m1 m2 jan mar
1) "mar" redis.sunion("m1", "m2")
2) "jan" mar <= ["mar", "jan", "feb"]
3) "feb"
jan

feb


Sorted Sets
zadd (add with score)

Redis REPL Groovy

> zadd z1 1 jan redis.zadd("z1", 1, "jan")
z1 1 jan
(integer) 1 <= 1


Sorted Sets
zscore (score for member)

Redis REPL Groovy
1 jan

z1 2 feb

> zscore z1 feb 3 mar redis.zscore("z1", "feb")
"2" <= 2.0

2


Sorted Sets
zrange (sorted subset)

Redis REPL Groovy
1 jan

z1 2 feb
> zrange z1 0 1 withscores
1) "jan" 3 mar redis.zrangeWithScores("z1", 0, 1)
2) "1" <= [["jan", 1], ["feb", 2]]
3) "feb"
4) "2"
1 jan

2 feb


Sorted Sets
zrangebyscore (subset having score range)

Redis REPL Groovy
1 jan

z1 2 feb
> zrangebyscore z1 2 3 withscores
1) "feb" 3 mar redis.zrangeByScoreWithScores("z1",2,3)
2) "2" <= [["feb", 2], ["mar", 3]]
3) "mar"
4) "3"
2 feb

3 mar


Groovy Usage


Grape @Grab Annotation

#! /usr/bin/env groovy
@Grab('redis.clients:jedis:2.0.0')

def redis = new redis.clients.jedis.Jedis("localhost")

assert "PONG" == redis.ping()


Producer/Consumer
Example


Producer
pushes work on a list with lpush


redis = new redis.clients.jedis.Jedis("localhost")

args.each { redis.lpush("welcome-wagon", it) }


Consumer
uses blpop (blocking left pop from list)



println "Joining the welcome-wagon!"

while (true) {
def name = redis.blpop(0, "welcome-wagon")[1]
println "Welcome ${name}!"
}


Mass Producer
srandmember to randomly pick female name from set


if (!redis.exists("female-names")) {
new File("./female-names.txt").eachLine {redis.sadd("female-names",it)}
}

for (i in 1..100000) {
redis.lpush("welcome-wagon", redis.srandmember("female-names"))
if (i % 1000 == 0) println "Adding $i"
}

female-names.txt from: http://antirez.com/post/autocomplete-with-redis.html

Groovy Demo


Grails Redis Plugin
https://github.com/grails-plugins/grails-redis


Plugin Config
in Config.xml

grails {
redis {
poolConfig {
// jedis pool specific tweaks here, see jedis docs & src
// ex: numTestsPerEvictionRun = 4
}
port = 6379
host = "localhost"
}
}


Provides Caching Through
Memoization


RedisTagLib

<redis:memoize key="mykey" expire="3600">

</redis:memoize>

RedisService
Spring bean wraps pool connection

// overrides propertyMissing and methodMissing to delegate to redis
def redisService

redisService.foo = "bar"
assert "bar" == redisService.foo

redisService.sadd("months", "february")
assert true == redisService.sismember("months", "february")


RedisService
template methods manage pooled Redis connection

redisService.withRedis { Jedis redis ->
redis.set("foo", "bar")
}


RedisService
template methods manage pooled Redis connection

redisService.withRedis { Jedis redis ->

redisService.withTransaction { Transaction transaction ->
transaction.set("qux", "baz")
assertNull redis.get("qux")
}

assertEquals "baz", redis.get("qux")
}


RedisService
String memoization

redisService.memoize("my-key") { Jedis redis ->
// expensive operation we only want to execute once
}

def ONE_HOUR = 3600 // with optional timeout in seconds
redisService.memoize("my-key-with-timeout", ONE_HOUR) { Jedis redis ->
// expensive operation we want to execute every hour
}


RedisService
Domain Class memoization (stores IDs hydrates from DB)

def key = "user:$id:friends-books"

redisService.memoizeDomainList(Book, key, ONE_HOUR) { redis ->
// expensive process to calculate all friend’s books
// stores list of Book ids, hydrates them from DB
}


Example
Showing Products with Sort/Filter/Pagination Criteria


Other Memoization Methods
memoizeHash, memoizeHashField,
memoizeScore (sorted set score)


Grails Redis-GORM Plugin
http://grails.github.com/inconsequential/redis/


Uses SpringData to abstract
data layer


Can be used in conjunction
with Hibernate


Partial support for GORM
including Dynamic Finders, Criteria, Named Queries and “Transactions”


Limitations
It requires explicit index mapping on fields you want to query

package com.example

class Author {

String name

static mapWith = "redis"
static hasMany = [books: Book]

static mapping = {
name index:true
}
}

Under The Covers
MONITOR output for new Author(name: "Stephen King").save()

1308027697.922839 "INCR" "com.example.Author.next_id"
1308027697.940021 "HMSET" "com.example.Author:1" "name" "Stephen King" "version" "0"
1308027697.940412 "SADD" "com.example.Author.all" "1"
1308027697.943318 "SADD" "com.example.Author:id:1" "1"
1308027697.943763 "ZADD" "com.example.Author:id:sorted" "1.0" "1"
1308027697.944911 "SADD" "com.example.Author:name:Stephen+King" "1"


Questions?


Redis & Groovy integration for caching and messaging

Recommended

Recommended

More Related Content

Recently uploaded

Recently uploaded (20)

Featured

Featured (20)

Redis & Groovy integration for caching and messaging