Zookeeper C# Client with IKVM .NET

September 27, 2014 Leave a comment

Why ?

A lot of big data technology created with JVM. The language are vary from plain old Java, Scala and Clojure. There are a huge number of Apache project for this kind of stuff. The infrastructure for building a serious data product is amazingly mature in Java world.

The server is in Java . Ok. fine. But do we have to code in Java too ? At least as a client using the server ?. I know that JVM is awesome, but the language is not so well compare to C#. Especially the latest one C# 5. But for now we have to admit that .NET world is very lack of Server solution.

I always try to find a client library in C#/.NET but seems that it’s not really mature or has been already dead before birth. I will give you an example for this one.

Zookeeper is for the distributed zoo

Zookeeper has been used as a distributed process control. If you still remember the operating system 101 about process, threads, inter-process communication so this one should be easy for you to learn. If you familiar using lock or any synchronization mechanism like mutex, semaphore etc, just think this as way to do to coordinate multiple server synchronization. And it’s only expose a simple primitive like a file system. This is amazing and flexible. At the same time also make you do a lot of works to get the stuff working.

image

So we know that Zookeeper is good, it used by a lot of big data technology like Storm, Kafka, etc. So we need to find a client library to use it right ? Good luck finding it for .NET. There are a couple of library for doing that stuff but it’s either old or not up to date.

A couple of sample of that kind of library is :

https://www.nuget.org/packages/ZooKeeper.Net/

https://www.nuget.org/packages/ZooKeeperNet/

https://github.com/devhong/Zookeeper.Net

Curator

If you try to find the similar library in Java you will be found Curator. It’s really a nice stuff. It implement a couple of recipes on Zookeeper. So you don’t need to scratch your head with an odd zookeeper API. It’s really make your eyes hurts. Especially if you are a C# programmer.

image

 

So how about Curator in .NET ? Is that really a dream. Actually no. You can see the similar stuff build in .NET with the previous nuget link I gave you. Especially this one .

https://github.com/ewhauser/zookeeper/tree/trunk/src/dotnet

image

But if you look at the last update time it’s really sad.

IKVM

After searching and searching. I found an interesting stuff about how to convert the java library into .NET library. I know this a long time ago. I seems to forget this one. It’s doing bytecode to IL translation. Isn’t that awesome ? It’s open source and free.

Please enter IKVM .NET

http://www.ikvm.net/

You can find the latest binary usually posted on the website. The sourceforge and the nuget seems haven’t been updated quite sometimes.

So go to this website http://weblog.ikvm.net/ and then find the latest ikvmbin. It also covers the lambda stuff from Java 8. At the time this blog post created I can see the ikvmbin 8.xx. Go download and extract.

image

Here’s a couple of video that discuss about IKVM.

Jeroen Frijters live in Action

http://channel9.msdn.com/Events/Lang-NEXT/Lang-NEXT-2012/IKVM-NET-Building-a-Java-VM-on-the-NET-Framework

And here’s an interesting conversation between two Dutch man. Smile

http://channel9.msdn.com/Blogs/Charles/E2E-Erik-Meijer-and-Jeroens-Frijters-IKVMNET-Java-VM-implemented-in-NET

Enough said show me the code..

Before we are working IKVM I think we need to install maven in our system. This will make us easier to maintain and download java library and the dependencies so we don’t need to download it ourselves.

Please download maven first from this website.

http://maven.apache.org/download.cgi

image

Also find the step by step to install that stuff under below the download section of the previous page.

To test your Maven installation you can follow this tutorial.

http://maven.apache.org/guides/getting-started/maven-in-five-minutes.html.

IKVM in Action

Make sure that you have already extract the IKVM and set your environment path variable to it’s bin directory. Also set up the environment path for Maven.

Now we need to download the Curator library with maven as that is easier because it has already contains some recipes.

Create a folder and pom.xml in that folder. Enter this config to pom.xml

image

Go to that folder and execute this command

mvn install dependency:copy-dependencies

It will download Curator and the direct dependencies.

Ok don’t be afraid to the bloated text/log in the console. If you come from .NET maybe you think that’s an error. Don’t worry. It’s Not. That’s just how it’s work. You will get use to it. Open-mouthed smile

You will find all the library/jar in target/dependencies folder

image

 

Ok now you are about to enter the parallel universe. Java and .NET with IKVM.

It’s just one line of command.

ikvmc -lib:target\dependency -recurse:target\dependency -target:library -out:Curator.dll

That command will convert all library in target\dependency and also figure out the dependency itself. The result will be output as Curator.dll.

You may see a couple of warning. You can fix that warning by adding the required dependencies into pom.xml.

image

 

Now you can find your familiar .NET dll library.

image

 

Let’s try that on your project. Don’t forget to start your Zookeeper server first. Find the step by step to start here

Create a config first in conf/zoo.cfg. Just copy the zoo_sample.cfg for now to get start quickly.

image

Then start your zookeeper server.

image

You can try to connect with commandline to your zookeeper server

image

 

 

Now create a new project on your beloved Visual Studio and include the Curator.dll. Also don’t forget to include all IKVM library. This will be required for Curator.dl

image

 

Create a console application and create a class to demonstrate the distributed lock. You can take a look this page on how to get there.

http://curator.apache.org/getting-started.html

 

image

On your main program enter this code.

image

Run two console application for this same project. You will see something like this. The first console will acquire the lock and the other one will wait.

image

 

After the first one finish with it’s work then it will release the lock and give another console to proceed with it’s task.

image

 

Curator is really Awesome right ? IKVM also great.

 

Conclusion

Very easy right ? Meh. I lied.

 

Converting the library is not so easy. You sometimes have to figure out the dependency that missing, add to pom.xml and download and try again.

Also not all library can be converted with IKVM especially the one that use a lot of magic stuff and undocumented library. For example IMDG GridGain. It use Unsafe class from sun.misc. for changing the value of public final field in interface. How crazy this is right ?

Seems that Unsafe class hasn’t been implemented completely in IKVM. Another class that I notice missing is the implementation of JMX on monitoring java system. Some method is not implemented.  So sometimes it works and sometimes it don’t so take your poison.

 

A little bit of advice

Another thing to consider is to wrap your java library implementation into wrapper so it will feel natural to .NET developer, you may need to convert a callback stuff using Task from TPL, also using async await and even Reactive Extensions.

Don’t let the java class implementation leak into your .NET code. Tha’s just not right. It hurts my feeling. So don’t use it directly , always use anti corruption layer on top of the converted library.

You maybe find this article is interesting too on how to working with converted library.

http://www.palladiumconsulting.com/2014/06/using-zookeeper-from-net-ikvm-task/

 

Bring that library to .NET !

 

Cheers

Head First Akka with Scala

August 1, 2014 Leave a comment

Howdy

Have you enjoy Scala yet ? Have you love this language and its actor model yet ?

Probably you have. I will now explore a really cool framework for building distributed system with actor model. It’s Akka Framework. It’s a reactive framework.

 

image

 

If you haven’t heard about the term reactive you need to check out this one first.

image

 

First we will need to install SBT first. If you have already familiar with java. Managing dependency, installing library, setting up structure is really painful if you are not using tools or IDE. SBT is a simple build tools that doesn’t sucks like Ant (with XML hell). You need to download that first from here.

image

Now we can start by creating SBT project  for this lesson. Create a file build.sbt and insert the following content into the file.

 

image

Run sbt compile for the first time for initializing sbt and your project. It will download the required library for Akka. Your output might be different with me. At the first time it will take awhile. Just be patience.

image

 

Create directory src/main/scala

 

image

Create file HelloActors.scala. Insert the following code

image

Run the project by executing sbt run from the directory where build.sbt created.

 

image

 

Now you can continue exploring by download the following chapter and write some code !!.

Cheers

 

References :

http://www.reactive.io/tips/2014/03/28/getting-started-with-actor-based-programming-using-scala-and-akka/

http://doc.akka.io/docs/akka/2.3.4/intro/getting-started.html

http://doc.akka.io/docs/akka/2.0/intro/getting-started-first-scala.html

http://doc.akka.io/docs/akka/2.0/index.html

http://doc.akka.io/docs/akka/2.3.4/scala/index-network.html/

http://doc.akka.io/docs/akka/2.3.4/scala/cluster-usage.html#preparing-your-project-for-clustering

Tools :

http://www.jetbrains.com/idea/features/scala.html

http://confluence.jetbrains.com/display/IntelliJIDEA/Scala?_ga=1.109172037.1933020012.1406117705

Categories: Uncategorized

Head First Scala

August 1, 2014 Leave a comment

It’s been a long time since I learn my last programming language. I’ve been using C# for about 6 years now. I think it’s time to learn something new.

Because of my current job heavily involved with concurrency and network I choose a language that can give me great advantage for that problem. There’s no doubt that JVM is the most reliable and mature runtime in the world. A big internet company like google, amazon and linkedin use it.

The great thing about that is there are a lot of library out there when you want to write a software. Just do your research, google it and you are ready to go with the short amount of time. The problem is the Java language become old and old. It’s not fun using that anymore.

Fortunately a bunch of smart people write another JVM based language like scala, clojure, groovy, erjang etc. This is great stuff, because you can have a taste of modern language while enjoying productivity using existing Java library.

I choose scala to learn first from a bunch of JVM languages. That’s because it’s have a multi paradim programming like functional, object oriented and concurrent. Ok. Let’s get started.

The first thing you want to do is installing JDK. You can install JDK 7 or latest. Download it from the official website.

image

After you install the JDK make sure that you set your environment variable. Follow this instruction.

 

Installing scala for Windows is very easy. Just download the MSI installer from here and run it.

image

 

Now open Command Line and type scala. Print hello world to console and celebrate your victory.

image

 

Learning a new language can be boring if you have already knew a lot from your previous experience. One approach is to visit the fundamental and unique characteristic from this language first and then going deep. One book I found very interesting for introducing the language with different paradigm is Seven Languages in Seven Weeks.

image

Be careful this is just an appetizer to bring the appetite. Smile with tongue out. After you buy the book then open Chapter 5 about Scala, read and type the code. You will amaze how fast you can learn this language. This language is very interesting because of it’s actor model. Also it will introduce you to functional concept like pattern matching, high order function.

 

Enjoy your first journey with Scala

Cheers

Categories: Uncategorized

Cassandra, C# and async await.. Oh my

June 11, 2014 Leave a comment

 

Cassandra is really cool stuff. Especially for time series. Even though  you are new to NOSQL the transition is pretty fast once. But first of all you need to learn to unlearn what you know before about relational database.

You need to download it first from DataStax. You can find the step by step installation from here

A couple of things you need to keep in mind is this principle :

 

  1. Don’t model your data. Model your query.
  2. Don’t afraid of data duplication
  3. Denormalization is Good

You also need to be familiar with Brewer’s CAP theorm and BASE.

Ok. Enough talking. Now let’s get down to code.

Cassandra is cool, so we need to find the programming language that’s totally awesome to access it. Smile 

C# it is !!

 

You can easily find the Cassandra library from Nuget. It’s DataStax library too.

 

image

 

The problem is of course the documentation. You can find the sample code from github project actually. Also from here. But dude… it’s still using TPL and ContinueWith callback model.

image

 

Can you imagine if you would try to execute that from loop ? Sad smile

Anders Helsjberg would cry for that one.

What is so hard about changing that one into this one. Everyone will rejoice about that.

image

 

Also I think DataStax need to support Reactive Extensions for their library. Instead of enumerable row it should return Observable.

I saw the another library that support TPL, Async Await and Rx from Nuget.

image

You can found it’s github here. I will definitely check that out later.

 

Until then please spend your time to understand concept, data modeling and architecture in Cassandra. Please read the following books. It’s the latest book that I think will safe you some time when you have a trouble with Cassandra.

imageimageimage

image

 

Have a great time with the pretty Cassandra.

 

Cheers

Categories: Uncategorized

Fix Octave 3.2.4 Plotting Hang Problem in Windows 8.1

November 8, 2013 4 comments

 

For the last few weeks I’ve been following Machine Learning from Coursera. It’s a very awesome website. It’s also FREE.

I learn Machine Learning because that’s a new area that seems interesting in this Big Data World. Until now I think totally our job as software developer is to create application that save our customer data and rarely analyzing it. Of course we provide report but I don’t think that’s enough nowadays.

Machine Learning course use Octave as a programming tools. You can think that as a free and open source MATLAB like program without User Interface. . You can do a lot a computation and prototyping faster there compare to the common programming language as c# or java. But I think for production you need to turn that into real programming language.

The problem I have with Octave it’s a Console Apps. It’s really sad for developer that have been used to Visual Studio for a long time. But we can do plotting from there. That’s really awesome and awkward. Smile But its fun.

The problem I have with running Octave in Windows 8.1 is the plotting windows hang after showing. I search the solution in the internet and finally find two line that solve that Hang problem. Seems that’s a library conflict in Octave

Just need to run this command on Octave CLI.

pkg rebuild -noauto oct2mat

pkg unload oct2mat

 

Reference

http://octave.1599824.n4.nabble.com/plot-window-hangs-td1681494.html
http://stackoverflow.com/questions/12032494/octave-always-not-responding-freeze-when-use-plot

Build on Windows, Deploy on Linux

May 14, 2013 1 comment

Yups.

I think there’s nothing can beat Visual Studio when it comes to .NET development. MonoDevelop like Miguel de Icaza said in this presentation is nothing close to VS2012. But I do interested on where Xamarin Studio will headed.

Developing and Compiling I think still ok on Windows. Just need to deploy and test an assembly on Linux.

So how we do that ? How we transfer that binary to the Linux server. I assume that the server won’t have a UI for performance sake. I think that’s the best. We can fine tune the server for processing concurrent request and not for the fancy UI.

The tools we need is just a simple binary executable and command on Windows. I know this from this blog post.

How to: Copying Files from Windows to Linux via SSH (vice-versa)

We need to download that binary from this page. Just find pscp.exe. Select the latest version. Download it directly.

 

image

As soon as it arrived on your windows machine, just copied it to Windows, so we can execute it from anywhere via command line.

You can use it like this.

pscp C:\Documents\helloWorld.txt username@165.1.122.76:/home/your/destination/folder/

Here’s how I test it

image

And I’m so happy when I see that file is in the server.

image

When you have a lots of files, it’s best to compress it so we can transfer it as a single file. You can use the default compression in windows.

image

Or you can use the 7zip if you are prefer the smaller files.

In this step I will use default windows compression. To unzip it in Linux you need to install it first. Just execute this command to install

yum install unzip

Then execute this command to unzip the file

unzip pics.zip  -d /tmp

 

Very easy right ?

 

Develop on Windows. Deploy on Linux

 

You will got the best from both worlds

 

Cheers

Categories: Uncategorized

FxSharp going Mono

May 10, 2013 Leave a comment

In this blog post I will tell you my experience porting my open source .NET project to Mono. It’s very easy.

FxSharp is my Expert Advisor System for MetaTrader 4 written in C#. MetaTrader is terminal used by Forex retail trader to enter the transaction. There’s a programming language called MQL. You can use it to create your own expert advisor, but it’s very ancient. Procedural and sucks ! MQL 5 is not a big different with MQL 4. It’s a object oriented language using c++ like syntax. It’s not c++. It’s only like c++. You can see a horrible codes written in this language. C# is better for developer productivity. I use Trade Platform .NET to bridge the MQL into C#. It’s very nice open source project.

First, we need to check out the source code from github.

image

Change to the directory and you will find solution file. We will build this with xbuild. Before that don’t forget to change the permission of that source code.

image

After xbuild the solution, you will find that everything is compiled fine.

image

image

But when I run it, i ‘ve got this simple and strange error message.

image

I’m very confused at the first time. Because it doesn’t tell the detail. So I do some googling and found that’s I’m using unsupported API. How do I know which one ?? Finally I found a nice tools called MoMA (The Mono Migration Analyzer)

I download and run it to analyze my application on windows.

image

Include all the assembly tobe analyzed. You can find guide using MoMA here.

image

I take a look at the report and found that there’s a problem in entry point.

image

Seems I use the spesific windows functionality here.

image

Well, I just need to comment it in the caller.

image

I run it again and voila. It’s works. My server is running on linux.

image

Ok. Now I need to connect the Expert Advisors to point to the Linux Servers. First I need to disable the firewall. I don’t want to get caught up setting one by one, so I choose to disable the firewall.

 

image

image

image

Fill the expert advisors settings. And Ok.

image

And… Boom. The program is running but seems it can’t load the assembly required for the process. Trade Platform load the assembly dynamically to App Domain. So there’s something wrong with the loading process I think. Why it can’t find the assembly…

image

Finally I found the problematic codes for loading assembly. It use hardcode to create path for the assembly.

image

I found a good explanation about the best practice for compatibility between windows and linux. Read this article.

 

image

I change the code into something like this. Using method .NET provided.

image

After push and pull from github. Running it. It’s working now !

The Quarts scheduler, parser, tcp/ip. All the functionality.

image

 

It’s very easy to run your .NET application on Linux. Mono is the answer.

It’s awesome

 

Cheers

Categories: Uncategorized
Follow

Get every new post delivered to your Inbox.

Join 1,568 other followers