Christian: Thanks so much for being involved with the Heron community. Can you talk a little bit about how you ended up getting involved?
Josh: I’m doing some work currently for a finance company that needed support for high-velocity transactions with very low latency. They are doing a lot of volume, so they need a stream processing system that can handle all that. As I was investigating possible solutions I found Apache Storm. It has an API called Flux that allows developers to modify topologies (pipelines of data) without the need to recompile with every modification, which was a key feature we needed. I kept looking for other solutions as well. That’s when I came across Heron which was developed at Twitter as a replacement for Storm. Heron is much more modern with a nice UI, and the management of topologies has been improved a lot. Also, I found the community to be extremely responsive to any questions I had, which is a great sign with an open-source project.
Christian: And so how did you decide to get involved with contributing to Heron?
Josh: Well, as I mentioned, support for the Flux-like capabilities in Storm was key for us, and unfortunately Heron didn’t have similar functionality implemented. I reached out to the community and pitched the idea about adding it, and they came back and said: “Great idea! Why don’t you write it?”
Christian: That’s the great thing about open source. If you have a need for a feature you can develop and contribute it and then the whole community benefits.
Josh: Exactly. I cleaned up the code, refactored it, and developed the ECO API for Heron. The committee picked up my pull request and merged it and now it is available in Heron. I didn’t add all the functionality yet as it’s important to get community feedback on features to know what to add next. You can read more about it in my post.
Christian: Thank you so much for contributing. What’s next for you and Heron?
Josh: To be honest I am most excited about seeing how it gets used. When you build software you tend to have a very egocentric view of it because you’re writing it for your own purpose. But then when other people pick it up they use it in ways you haven’t even thought of. For example, Karthik, one of the creators of Heron, is already using it to write different connectors. I think that’s pretty cool.
Christian: Maybe we can interview you again in a few months and see how it is being used and what else you’ve learned about Heron by then.
Josh: I’d be happy to!
Josh Fischer is an engineer at 1904Labs in St. Louis, MO who has worked on some impressive projects for major organizations. He recently authored a post about his work developing ECO for the Heron stream processing engine and I caught up with him recently for a chat about how he got involved in the community.