We are from Geekseat has been used Avro extensively. Currently our project needs to be done on .NET core so .As i post previously i’m working on porting Apache Avro into .NET Core. Basically there are two Avro library available. One from Apache and one from Microsoft. I want to port both of library as that has been used a lot in big data application.
Let’s get back to Apache Avro.
There are two main problem that made headache when porting this one. One is AppDomain and IL Generator / Dynamic method is missing.
Both problem exists in the ObjectCreator.cs. This class is responsible for dynamically creating object on the fly based on the avro type.
ILGenerator has been known as the fastest object creator by Ayende Research. So i think this can be replaced with Expression tree but not sure how. Then i remember that Zeddy Iskandar has been create a forum post on how to create object with expression tree with constructor. You can see the method from this forum post. Here’s the complete source code of the ctor delegate.
One down, one more to go.
I use the polyfill approach from that blog post. You can see that in action here.
After fixing both problem i’ve got 382 test passed and 6 failing. I’m quite happy with this result, declare victory and push that to repository. Thanks Zeddy and Michael.
You can follow the tutorial for publishing your library from this following series of youtube videos. You will need to request via Sonatype JIRA and wait for approval. It’s explained in the series.
However if you want the easiest way you can leverage using bitbucket pipeline. You can see the step by step on how to do that from this tutorial. It’s still in beta phase but looks promising. You should give it a try and ask for the trial. Yes, you need to request the trial first and wait for the approval.
The missing part on the tutorial above is on how to generate the key. Here’s how you can do that in Windows machine.
- Download the gnugpg for windows here
- Setup the key for signing
- Generate the key
- Find the list of the key to find the id
Let’s say that the id is 4C7CF393. We will use this key in the subsequent step.
- Publish the public key
gpg2 –keyserver hkp://pool.sks-keyservers.net –send-keys 4C7CF393
- Export the private key
gpg -a –export-secret-key 4C7CF393 > private-key.gpg
- Donlot http://gnuwin32.sourceforge.net/packages/openssl.htm for encryption
- Add the binaries to your PATH environment variables
- Encrypt the key
openssl aes-256-cbc -in private-key.gpg -out private-key.gpg.enc -pass pass:somesecretpwd
- Generate the key
- Upload that key to your repository. You can base your repository by copying this repository and replacing the private key there.
- Copy your source code and etc there.
- Replace the configuration for username and password by looking at README from this repository
- Edit the pom.xml so it publish your library
- You can follow the rest of the tutorial from this
Happy Publishing !
Finally ETP java library available in maven central. You could use this one in your pom.xml
<dependency> <groupId>org.energistics</groupId> <artifactId>etp</artifactId> <version>1.4.1</version> </dependency>
Here’s the sample of my pom.xml
It will automatically import avro as dependency. Now you can use it from your project.
You can also find the various way to add that to your build system from this link
All the source code and pom.xml that required to generate the java source using maven plugin can be seen from ETP Java Bitbucket
And you can find the pom example in my Github
.NET core supports three unit test framework in this version. We can also use Test Explorer in Visual Studio. Here’s the configuration in project.json
Here’s three project with each test framework and a couple of test class
Open the test explorer from Menu Test -> Window -> Test Explorer
Please build the test projects first and the test will be shown in Test Explorer.
You can access the source code and configuration in my Github
Happy TDD-ing !!