Building dynamic distributed data stores with OSGi - Tim Ward

Building dynamic distributed data
stores with OSGi
http://www.paremus.com
info@paremus.com

Tim Ward
Dynamic Distributed Data Stores in OSGi

Wednesday, 30 October 13

Copyright © 2005 - 2013 Paremus Ltd.
May not be reproduced by any means without express permission. All rights reserved.

Oct 2013

Who is Tim Ward?
@TimothyWard

• Senior Consulting Engineer and Architect at Paremus
• 5 years at IBM developing WebSphere Application Server
• Container Implementation experience with Java EE and OSGi, including Blueprint, JPA, EJB
and JTA

• OSGi Speciﬁcation lead for JPA and Bytecode Weaving
• PMC member of the Apache Aries project
• Previous speaker at EclipseCon, Devoxx, Jazoon, JAX London, OSGi
Community Event...

• Author of Manning’s Enterprise OSGi in Action
• http://www.manning.com/cummins



Oct 2013

What we’re going to cover

• OSGi Remote Services
• Deploying external processes using Packager
• Building Dynamically wired distributed systems
• Demo




Oct 2013

OSGi Remote Services




Oct 2013

OSGi Remote Services (1)




Oct 2013


• OSGi Bundles communicate using the Service Registry




Oct 2013


• This allows for dynamic, loosely coupled interaction
Provider A

Consumer




Oct 2013


• This allows for dynamic, loosely coupled interaction
Provider A

Consumer

Provider B




Oct 2013





Oct 2013


• Because OSGi services are loosely coupled they can come from anywhere!




Oct 2013


• Any service can be transparently remoted *




Oct 2013


• Service properties are made available remotely too




Oct 2013


Network
Provider A

Wormhole




Oct 2013


Network
Provider A

Consumer

Wormhole




Oct 2013


Network
Provider A

Consumer

Wormhole
* Make sure your service is actually suitable for remoting!



Oct 2013





Oct 2013


• Exposing a remote service is not transparent




Oct 2013



Network

objectclass =
{List, Collection}
Provider A

Consumer

Wormhole




Oct 2013


• Services opt-in using the service.exported.interfaces property

Network

objectclass =
{List, Collection}
Provider A

Consumer

Wormhole




Oct 2013



objectclass =
{Collection}

Network

objectclass =
{List, Collection}
Provider A

Consumer

Wormhole



service.exported.interfaces =
{Collection}


Oct 2013


• Imported services are marked with a service.imported = true
objectclass =
{Collection}

Network

objectclass =
{List, Collection}
Provider A

Consumer
service.imported =
true



Wormhole

service.exported.interfaces =
{Collection}


Oct 2013

OSGi Remote Services and Remote Service Admin




Oct 2013


• The OSGi Alliance has two speciﬁcations. Remote Services and Remote
Service Admin (RSA)




Oct 2013


Service Admin (RSA)

• What I have described is Remote Services




Oct 2013


Service Admin (RSA)

• The underlying implementation is completely unspeciﬁed




Oct 2013


Service Admin (RSA)

• RSA builds on Remote Services, and is more prescriptive




Oct 2013


Service Admin (RSA)

• Pluggable distribution mechanisms (WS, RMI, JMS...)




Oct 2013


Service Admin (RSA)

• Pluggable discovery mechanisms (SLP, mDNS, Zookeeper...)




Oct 2013


Service Admin (RSA)

• Pluggable discovery mechanisms (SLP, mDNS, Zookeeper...)
• Notiﬁcations for “Topology Manager” bundles to decide which services to
export/import




Oct 2013

Packaging Applications with Packager




Oct 2013

Packager Aims and advantages




Oct 2013


• Package programs as self-describing OSGi Bundles




Oct 2013


• Allows Dynamic Resolution and Assembly of runtimes




Oct 2013


• Can be semantically versioned




Oct 2013


• Leverage the existing bundle/service lifecycle and API




Oct 2013


• Allow processes to be installed and uninstalled in a running system




Oct 2013


• Allow processes to be dynamically started and stopped




Oct 2013


• Allow integration with other OSGi speciﬁcation




Oct 2013


• Dynamic Conﬁguration using Conﬁg Admin,




Oct 2013


• Use and provide Local and Remote Services




Oct 2013


• Use and provide Local and Remote Services
• Allow existing code to be packaged without change



Oct 2013

Approach




Oct 2013

Approach

• Wrap Native Artifacts in OSGi Bundles




Oct 2013

Approach

• Link the Artifact Lifecycle to OSGi.




Oct 2013

Approach

• Bundle Install/Resolve/Start



=> Artifact “install”


Oct 2013

Approach

• Bundle Install/Resolve/Start => Artifact “install”
• Service registration/unregistration => Artifact “run/stop”




Oct 2013

Approach

• Bundle Uninstall => Artifact “uninstall”




Oct 2013

Approach


• Link to Standard OSGi Services: Conﬁguration Admin, Metatype, Log
Service...




Oct 2013

Approach


• Link to Standard OSGi Services: Conﬁguration Admin, Metatype, Log
Service...

<system name="BackEnd:MongoDB" boundary="fibre">

<system.part category="msf" name="com.paremus.packager.demos.mongo.guard">
<property name="port" value="27017" />
</system.part>
</system>



Oct 2013

Separation of Concerns




Oct 2013


• What are we running, and how do we conﬁgure/run it?




Oct 2013


• Package Type Service




Oct 2013


• When should we start it, and what conﬁguration properties should we use?




Oct 2013


• Process Guard Service




Oct 2013


• Actually invoke the start/stop scripts, and monitor the state of the external
process




Oct 2013


• Actually invoke the start/stop scripts, and monitor the state of the external
process

• Packager Manager/Watchdog Process



Oct 2013

Packager Package Types




Oct 2013

Package Type




Oct 2013

Package Type

• Usually contains the native parts




Oct 2013

Package Type

• Installs the Native Program




Oct 2013

Package Type

• Conﬁgures the program based on the supplied conﬁg




Oct 2013

Package Type

• Returns scripts to the Packager for: start, stop, ping...




Oct 2013

Package Type

• Is (usually) platform speciﬁc




Oct 2013

Package Type

• Is (usually) platform speciﬁc
Provide-Capability: packager.type;
packager.type=mongodb;
version:Version=2.2.0
Require-Capability: osgi.native;
filter:=”(&
(osgi.native.osname=Linux)
(osgi.native.processor=x86-64)
)”



Oct 2013

How the Package Type works




Oct 2013


• Installing an external process can take a long time




Oct 2013


• Not suitable for running in a bundle’s Activator!




Oct 2013


• Installation should be asynchronous with respect to bundle start




Oct 2013


• Run on a separate thread?




Oct 2013


• Install on demand?




Oct 2013


• Notify possible clients when a type is “ready” using the whiteboard pattern




Oct 2013


• Service properties advertise what the external process is
Package Type




Oct 2013


• Service properties advertise what the external process is
• Service unregisters to “uninstall”
Package Type




Oct 2013

Packager Process Guards




Oct 2013

Process Guard Service




Oct 2013


• Gathers Process Conﬁg Properties




Oct 2013


• Existence signals When to Start/Stop




Oct 2013


• Receives State Change Events (started, crashed...)




Oct 2013


• Advertises the Running Package (Optional)




Oct 2013


• Advertises the Running Package (Optional)
• Usually platform independent




Oct 2013

How the Process Guard works




Oct 2013


• Most external processes require some level of conﬁguration to run




Oct 2013


• This conﬁguration is often platform independent




Oct 2013


• When a process should be started the process guard registers a service




Oct 2013


• Another example of using the whiteboard pattern




Oct 2013


• It uses the same service properties as the package type




Oct 2013


• Often implemented as a ManagedService or ManagedServiceFactory




Oct 2013


• Automatically Inject/Delete the conﬁguration for the process
Process Guard




Oct 2013


• Automatically Inject/Delete the conﬁguration for the process
Config Admin



Process Guard


Oct 2013

The Packager Manager




Oct 2013


• Listens for matching Package Type and Process Guard Services




Oct 2013


• One Package Type may match many Process Guards




Oct 2013


• For every pair the packager manager starts an instance of the process




Oct 2013


• Conﬁgured by the Package Type with properties from the Process Guard




Oct 2013


• A ‘Watchdog’ monitors the process (using scripts from the package type)




Oct 2013


• Sends notiﬁcations to the Process Guard about the state of the process




Oct 2013


• Sends notiﬁcations to the Process Guard about the state of the process
• If either service is unregistered then the process is shut down



Oct 2013

Packager Interactions
Package Type

Packager Manager

Process Guard




Oct 2013

Package Type

Register services
Packager Manager

Process Guard




Oct 2013

Package Type

Get Configuration
Packager Manager

Process Guard




Oct 2013

Package Type

Packager Manager

Create Scripts and start!

Process Guard




Oct 2013

Package Type

Packager Manager

Monitor and notify

Process Guard




Oct 2013

Package Type

Packager Manager

Shutdown on unregistration

Process Guard




Oct 2013

Building dynamically wired systems with
Remote Services and Packager




Oct 2013

Publishing an Endpoint




Oct 2013


• How do you tell the rest of the framework that your external process
has started, or where it is?




Oct 2013



• The whiteboard pattern to the rescue again!




Oct 2013



• The Process Guard can register an “Endpoint” with a marker interface
and a service property with a URI to the process




Oct 2013




• It can also make this remotely available...




Oct 2013




• It can also make this remotely available...
• Suddenly the process is discoverable in every framework



Oct 2013

Dynamically building configurations




Oct 2013


• Process Guard services can be very simple




Oct 2013


• Static properties




Oct 2013


• Injection from Conﬁg Admin




Oct 2013


• They can also wait and listen for Endpoints (local and remote!)




Oct 2013


• Trigger a ManagedServiceFactory with part of the conﬁguration




Oct 2013


• Listen for the Endpoint, complete the conﬁguration




Oct 2013


• Listen for the Endpoint, complete the conﬁguration
• Register the Process Guard!



Oct 2013

Dynamically building configurations (2)




Oct 2013


• Replicating this out over multiple nodes lets you dynamically assemble
large systems with little to no manual conﬁguration input




Oct 2013



• Java EE servers




Oct 2013



• Java EE servers
• Messaging infrastructure




Oct 2013



• Java EE servers
• NoSQL stores




Oct 2013



• Java EE servers
• NoSQL stores
• Because OSGi services are dynamic we can track the service to see if
the endpoint moves!




Oct 2013



• Java EE servers
• NoSQL stores
• Because OSGi services are dynamic we can track the service to see if
the endpoint moves!

• Automatic reconﬁguration can rewire the system if a node fails!



Oct 2013

Putting Packager to the test with Redis




Oct 2013

What is Redis?




Oct 2013

What is Redis?

• Redis is an open source key-value store (http://redis.io)




Oct 2013

What is Redis?

• Redis supports queries based on key or value data




Oct 2013

What is Redis?

• Values can be rich objects like hashes, lists sets or sorted sets




Oct 2013

What is Redis?

• Redis supports replication to read-only “slave” nodes




Oct 2013

What is Redis?

• Redis supports replication to read-only “slave” nodes
• Wouldn’t it be nice to automatically conﬁgure the slaves?




Oct 2013

Demo




Oct 2013

Thanks!

• For more about OSGi...
• Speciﬁcations at http://www.osgi.org
• Enterprise OSGi in Action
• http://www.manning.com/cummins
• For more about Packager...
• http://www.bundlerepo.org:8090/display/SF110/Packager

Questions?



http://www.paremus.com
info@paremus.com
Oct 2013

Building dynamic distributed data stores with OSGi - Tim Ward

Recommended

Recommended

More Related Content

Similar to Building dynamic distributed data stores with OSGi - Tim Ward

Similar to Building dynamic distributed data stores with OSGi - Tim Ward (20)

More from mfrancis

More from mfrancis (20)

Recently uploaded

Recently uploaded (20)

Building dynamic distributed data stores with OSGi - Tim Ward