Subscribe Free

Join 2670+ others. No spamming.
I promise!

We are currently under high development. Follow us at github.


Looking for Python Tutorials?
Check these awesome tutorials



MichaelDrogalis/onyx

1051

MichaelDrogalis / onyx

Clojure

Distributed, masterless, high performance, fault tolerant data processing


READ ME

Logo Onyx

Join the chat at https://gitter.im/onyx-platform/onyx

What is it?

  • a masterless, cloud scale, fault tolerant, high performance distributed computation system
  • batch and stream hybrid processing model
  • exposes an information model for the description and construction of distributed workflows
  • Competes against Storm, Cascading, Cascalog, Spark, Map/Reduce, Sqoop, etc
  • written in pure Clojure

What would I use this for?

  • Realtime event stream processing
  • CQRS
  • Continuous computation
  • Extract, transform, load
  • Data transformation à la map-reduce
  • Data ingestion and storage medium transfer
  • Data cleaning

Installation

Available on Clojars:

[org.onyxplatform/onyx "0.10.0-beta3"]

Plugins and Libraries

Supported

Supported Plugins:

onyx-seq

onyx-seq is now included with onyx core. See onyx.plugin.seq and onyx.tasks.seq.

onyx-core-async

onyx-core-async has some breaking changes described in changes.

onyx-kafka

onyx-kafka should work without issues.

onyx-datomic

onyx-datomic should work without issues.

onyx-amazon-sqs

onyx-amazon-sqs should work without issues.

onyx-amazon-s3

onyx-amazon-s3 should work without issues.

onyx-bookkeeper

onyx-bookkeeper should work without issues.

onyx-metrics

onyx-metrics currently only supports JMX metrics reporting. There are some breaking changes with respect to use. See README

onyx-peer-http-query

onyx-peer-http-query now supports an onyx-metrics endpoint at "/metrics" which is compatible with prometheus. Simply enable onyx-metrics and your metrics will be reported to JMX.

onyx-dashboard

An 0.10.0 compatible onyx-dashboard release is available

Plugin Use

To use the supported plugins, please use version coordinates such as [org.onyxplatform/onyx-amazon-sqs "0.10.0.0-alpha2"], and read the READMEs on the 0.10.x branches linked above.

Currently Unsupported

Build Status

Component release unstable compatibility
onyx core Circle CI Circle CI -
onyx-kafka Circle CI Circle CI Circle CI
onyx-kafka-0.8 Circle CI Circle CI Circle CI
onyx-datomic Circle CI Circle CI Circle CI
onyx-redis Circle CI Circle CI Circle CI
onyx-sql Circle CI Circle CI Circle CI
onyx-bookkeeper Circle CI Circle CI Circle CI
onyx-seq Circle CI Circle CI Circle CI
onyx-durable-queue Circle CI Circle CI Circle CI
onyx-elasticsearch Circle CI Circle CI Circle CI
onyx-amazon-sqs Circle CI Circle CI Circle CI
onyx-amazon-s3 Circle CI Circle CI Circle CI
onyx-http Circle CI Circle CI Circle CI
learn-onyx Circle CI - Circle CI
onyx-examples Circle CI Circle CI Circle CI
onyx-dashboard Circle CI Circle CI Circle CI
onyx-metrics Circle CI Circle CI Circle CI
onyx-peer-http-query Circle CI Circle CI Circle CI
  • release: stable, released content
  • unstable: unreleased content
  • compatibility: edge, unstable, unreleased content depending on core master

Companies Running Onyx in Production

                                                                           

Quick Start Guide

Feeling impatient? Hit the ground running ASAP with the onyx-starter repo and walkthrough. You can also boot into preloaded a Leiningen application template.

User Guide 0.10.0-beta3

Developer's Guide 0.10.0-beta3

API Docs 0.10.0-beta3

Code level API documentation can be found here.

Official plugin listing

Official plugins are vetted by Michael Drogalis. Ensure in your project that plugin versions directly correspond to the same Onyx version (e.g. onyx-kafka version 0.10.0-beta3.0-SNAPSHOT goes with onyx version 0.10.0-beta3). Fixes to plugins can be applied using a 4th versioning identifier (e.g. 0.10.0-beta3.1-SNAPSHOT).

Generate plugin templates through Leiningen with onyx-plugin.

3rd Party plugin listing

Unofficial plugins have not been vetted.

Offical Dashboard and Metrics

You can run a dashboard to monitor Onyx cluster activity, found here. Further, you can collect metrics and send them to the dashboard, or anywhere, by using the onyx-metrics plugin.

Need help?

Check out the Onyx Google Group.

Want the logo?

Feel free to use it anywhere. You can find a few different versions here.

Running the tests

A simple lein test will run the full suite for Onyx core.

Contributor list

Acknowledgements

Some code has been incorporated from the following projects:

License

Copyright © 2015 Michael Drogalis

Distributed under the Eclipse Public License, the same as Clojure.

Profiler

YourKit

YourKit supports open source projects with its full-featured Java Profiler. YourKit, LLC is the creator of YourKit Java Profiler and YourKit .NET Profiler, innovative and intelligent tools for profiling Java and .NET applications.