Network analysis using Hadoop and Neo4j

Network analysis using
Hadoop en Neo4j

Friso van Vollenhoven
@fzk
fvanvollenhoven@xebia.com

ws
RT: @krisgeus...

follo
RT: @krisgeus...
RT

ows
RT

foll @fzk

RT
RT: @krisgeus...

A toy problem...

TABLE_DUMP2|1332345590|B|195.66.224.97|1299|1.11.64.0/21|
1299 6461 9318 38091|EGP|195.66.224.97|0|0||NAG||

AS1299 AS9318

AS6461 AS38091

TABLE_DUMP2|1332345590|B|195.66.224.97|1299|1.11.64.0/21
TABLE_DUMP2|1332345590|B|195.66.224.97|1299|1.11.64.0/21
TABLE_DUMP2|1332345590|B|195.66.224.97|1299|1.11.64.0/21
TABLE_DUMP2|1332345590|B|195.66.224.97|1299|1.11.64.0/21

raw
TABLE_DUMP2|1332345590|B|195.66.224.97|1299|1.11.64.0/21
TABLE_DUMP2|1332345590|B|195.66.224.97|1299|1.11.64.0/21

transform nodes.txt
TABLE_DUMP2|1332345590|B|195.66.224.97|1299|1.11.64.0/21
TABLE_DUMP2|1332345590|B|195.66.224.97|1299|1.11.64.0/21
TABLE_DUMP2|1332345590|B|195.66.224.97|1299|1.11.64.0/21
TABLE_DUMP2|1332345590|B|195.66.224.97|1299|1.11.64.0/21

data +
TABLE_DUMP2|1332345590|B|195.66.224.97|1299|1.11.64.0/21
TABLE_DUMP2|1332345590|B|195.66.224.97|1299|1.11.64.0/21
TABLE_DUMP2|1332345590|B|195.66.224.97|1299|1.11.64.0/21
TABLE_DUMP2|1332345590|B|195.66.224.97|1299|1.11.64.0/21

edges.txt
TABLE_DUMP2|1332345590|B|195.66.224.97|1299|1.11.64.0/21
TABLE_DUMP2|1332345590|B|195.66.224.97|1299|1.11.64.0/21
TABLE_DUMP2|1332345590|B|195.66.224.97|1299|1.11.64.0/21

en
TABLE_DUMP2|1332345590|B|195.66.224.97|1299|1.11.64.0/21
TABLE_DUMP2|1332345590|B|195.66.224.97|1299|1.11.64.0/21
TABLE_DUMP2|1332345590|B|195.66.224.97|1299|1.11.64.0/21
TABLE_DUMP2|1332345590|B|195.66.224.97|1299|1.11.64.0/21

ric
h
?
import enriched-nodes.txt
+
enriched-edges.txt

qu
int er
er y /
ac
t

[head]

GlobHfs[/Users/friso/Downloads/bview/alltxt.txt]

[{14}:'proto', 'time', 'type', 'peerip', 'peeras', 'preﬁx', 'path', 'origin', 'nexthop', 'localpref', 'MED', 'community', 'AAGG', 'aggregator'] [{14}:'proto',
[{14}:'proto', 'time', 'type', 'peerip', 'peeras', 'preﬁx', 'path', 'origin', 'nexthop', 'localpref', 'MED', 'community', 'AAGG', 'aggregator'] [{14}:'proto',

Each('nodes')[PathToNodes[decl:'id', 'name']]

[{2}:'id', 'name']
Each('edges')[PathToEdges[decl:'from', 'to', 'updatecount']]
[{2}:'id', 'name']

[{3}:'from', 'to', 'updatecount']
Each('nodes')[FilterPartialDuplicates[decl:'id', 'name']]

[{2}:'id', 'name']
GroupBy('edges')[by:['from', 'to']]
[{2}:'id', 'name']

edges[{2}:'from', 'to']
GroupBy('nodes')[by:['id']]

nodes[{1}:'id']
Every('edges')[Sum[decl:'updatecount'][args:1]]
[{2}:'id', 'name']

Every('nodes')[First[decl:'id', 'name']]

[{2}:'id', 'name']
Hfs['TextDelimited[['from', 'to', 'updatecount']]']['/tmp/edges']']
[{2}:'id', 'name']

Hfs['TextDelimited[['id', 'name']]']['/tmp/nodes']']

[{2}:'id', 'name']
[{2}:'id', 'name']

[tail]

http://bit.ly/IzWvcT and http://bit.ly/HHNNIb

nodes.txt:
1 AS1 LVLT-1 - Level 3 Communications, Inc.
10 AS10 CSNET-EXT-AS - CSNET Coordination and Information Center (CSNET-CIC)
100 AS100 FMC-CTC - FMC Central Engineering Laboratories
1000 AS1000 GONET-ASN-17 - GONET
10000 AS10000 NCM Nagasaki Cable Media Inc.
10001 AS10001 MICSNET Mics Network Corporation
10002 AS10002 ICT IGAUENO CABLE TELEVISION CO.,LTD
10003 AS10003 OCT-NET Ogaki Cable Television Co.,Inc.
10004 AS10004 AS-PHOENIX-J JIN Office Service Inc.
10006 AS10006 SECOMTRUST SECOM Trust Systems Co.,Ltd.
10010 AS10010 TOKAI TOKAI Communications Corporation
10011 AS10011 ADVAN advanscope.inc
10012 AS10012 FUSION Fusion Communications Corp.

edges.txt:
1 21616 3
1 3705 3
1 2 3
2 3 1
3 4 2
3 11488 2
4 5 1
10 10 2
10 13227 2
12 12 1

public class SillyImporter {
private static enum ConnectionTypes implements RelationshipType {
FOLLOWS;
}

public static void main(String[] args) {
BatchInserter database = new BatchInserterImpl("/Users/friso/Desktop/graph.db");
BatchInserterIndexProvider provider = new LuceneBatchInserterIndexProvider(database);
BatchInserterIndex index = provider.nodeIndex("allnodes", stringMap(
"type", "fulltext",
IndexManager.PROVIDER, "lucene"
));

long fzkNodeId = database.createNode(map(
new Object[] {"name", "fzk", "tweets", 25L} //node properties
));
index.add(fzkNodeId, map(new Object[] {"name", "fzk"}));

long krisgeusNodeId = database.createNode(map(
new Object[] {"name", "krisgeus", "tweets", 100L} //node properties
));
index.add(krisgeusNodeId, map(new Object[] {"name", "krisgeus"}));

database.createRelationship(
krisgeusNodeId, //from node
fzkNodeId, //to node
ConnectionTypes.FOLLOWS, //relationship type
map(new Object[] {"retweets", 3})); //relationship properties

index.flush();

provider.shutdown();
database.shutdown();
}
}

30M nodes + 250M edges, < 30 minutes
(if your graph ﬁts in memory)

Cypher graph query language

start a = node:allnodes(‘name:”fzk”’)
match p = a-[r]-b
return p

All relationships from node with name=fzk


start a = node:allnodes(‘name:”fzk”’)
match p = a-[r]-b
where any(x in r.amounts where x > 500)
return p

All relationships from node with name=fzk, where any
element in the array property amounts of the relationship
is greater than 500


start a = node:allnodes(‘name:”fzk”’),
b = node:allnodes(‘name:”krisgeus”’)
match p = a-[:FOLLOWS*1..3]-b
return p

All paths with a length between 1 and 3 (inclusive) between
node with name=fzk and node with name=krisgeus with
any relationship type=FOLLOWS


start a = node:allnodes(‘name:”fzk”’),
b = node:allnodes(‘name:”krisgeus”’)
match p = a<-[:FOLLOWS]-x-[:FOLLOWS]->b
return x

All people who follow both fzk and krisgeus. Note the
indication of direction in the relationship predicates.

Toy problem and viewer source code:
https://github.com/friso/graphs

Questions?
Friso van Vollenhoven
@fzk
fvanvollenhoven@xebia.com

Network analysis using Hadoop and Neo4j

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (20)

Similar to Network analysis using Hadoop and Neo4j

Similar to Network analysis using Hadoop and Neo4j (20)

More from fvanvollenhoven

More from fvanvollenhoven (6)

Recently uploaded

Recently uploaded (20)

Network analysis using Hadoop and Neo4j