Today, thanks to Erlang Solutions I have the honour to publish on my blog an interview with Ulf Wiger who will speak at the Erlang Factory 2013 in San Francisco. The interview will range from his current work experience at Feuerlabs to his previous experiences in Ericsson and Erlang Solutions, with some interesting curiosies.
Ulf Wiger’s talk at Erlang Factory will be “KVDB – A Database System for Configuration Data and Connected Devices“, you can find more information here.
Ulf doesn’t need any presentation. He is one of the most respected figures in the Erlang Community, so let’s go straight to the questions, the interview is quite long, I hope you will enjoy it.
Mirko: Hi Ulf, first of all thanks for your time. I am glad to have you here on my blog today. When we met in London 3 years ago you were Erlang Solutions’ CTO, now I have the opportunity to interview you but your position has changed. You are now co-founder at Feuerlabs. Talk to me about it, why have you decided to found your own company and what are you doing at Feuerlabs?
Ulf: I imagine that many people in the community saw me as a Big Company guy during my years at Ericsson, but I really started my career working in a very small company. I spent four years working out of a basement in Alaska. I learned a lot about the joys and hardships of working in a startup, and I’ve always longed to get back to it.
A combination of events triggered me to start considering opportunities outside of Ericsson, not least that Erlang had actually evolved into a respectable technology within the company. But I also felt that I had done most of the things I came there to do – a few times over in many cases. I was becoming a bit complacent, and I needed to find more challenges.
(The thing you learn in a big company too, is that being ‘indispensable’ means that it will take a few weeks at most before you’re replaced, should you decide to leave. Once you’ve really understood that, I think it’s time to move on).
Erlang Solutions was another type of challenge, and also a chance to help out my good friend Francesco. It was a great ride and a learning experience, but it also reminded me that what really makes me tick is to build things. I see myself as a tinkerer and inventor who’s spent a decade and a half learning about all that other stuff that goes on around us – administration, project management, marketing… But I’m ultimately a programmer at heart, and it was time for me to focus on that again.
Feuerlabs offered that opportunity. I’ve known Tony Rogvall since the early years at Ericsson, when he worked at CSLab and I was on the AXD 301 project. The thing about Tony was that whatever programming challenge you addressed with him, he had a prototype lying in a drawer somewhere – most of the packet decoding logic in the first AXD 301 grew out of such a prototype. We’ve stayed in touch since then, partly because Tony helped advise Erlang Solutions on Erlang for Embedded Devices, and partly because we both seem to enjoy coming up with wild ideas. Tony, whom I already knew as a superb hacker and serial entrepreneur, and his old friend Magnus Feuer had been trying to find an excuse to work together in a startup, as they had done in the past. I’ve learned from experience that if you want to be invited into a startup, you need to indicate clearly that you’re interested – otherwise people will assume that you’re not crazy enough to want in. So this time, I told Tony that I wanted to join, if possible.
I hadn’t met Magnus before, but we had a very good talk over the phone, and I had a chance to meet him the week before we decided to launch the company. That felt a bit unusual, but I trusted Tony’s judgement, and also took to Magnus immediately.
What I do at Feuerlabs? I cut code! We’ve divided responsibilities, and I ended up writing most of the code in the device management server. It was a pretty obvious choice, since both Tony and Magnus are great C programmers with device programming experience. I may be many things, but a great C programmer I am not. I do know a thing or two about developing server-side software, though.
Mirko: Reading on FeuerLabs website’s section “Who we are”, I found out you are not only the co-founder but even a “Developer Advocate” also. What does it mean?
Ulf: We envisioned from the start that I would draw on my community experience and do most of the talking at conferences and interaction with the developer community. So far, we haven’t done that much external communication, but we are getting ready to become much more visible now.
Mirko: Let’s talk about KVDB which is the subject of your talk at the San Francisco Erlang Factory. I don’t want to go too deep because you have an entire talk about it. But why a database? In the last years Databases and Data related technologies are growing like mushrooms. Where KVDB comes from and what needs it is going to address?
Ulf: Well, building a database management system has become a lot easier than it used to be.
First of all, we spent a lot of time thinking about the data representation on the server. This is an area where not least Tony has much useful experience – having pioneered automatic device configuration for mobile phones in a previous venture, Synapse, and seen the system grow to supporting many millions of devices.
We had the idea that we wanted to view the device-side database as a small subset of the server-side database. Also, since we were building both a device management service and a device-side framework, we had to choose between having two different databases, or one that could work well in both environments.
Mirko: Writing a Database from scratch means addressing a wide range of problems, from consistency to transaction support and other non-trivial things. You have some previous knowledge from Mnesia. How did it help you? Which is the most difficult thing you encountered while developing KVDB?
Ulf: Yes, I have done quite a lot of tinkering with Mnesia. So has Tony – at Synapse, they integrated Berkeley DB into Mnesia, and also developed ways to replicate huge amounts of data. In my hobby project, ‘rdbms’, I ended up implementing most high-level aspects of a relational database that were missing in Mnesia, and also did a lot of work on extensible indexing. Actually, ErlHive also ended up giving some useful experience working with hierarchical data structures.
The things that make KVDB a bit special are that it supports multiple backends (granted, Riak does that too), and that it’s somewhat specialized in dealing with hierarchical data structures, such as configuration data trees, registries, etc. It also supports a few unusual things, such as persistent queues, persistent timers, etc. To make this work well together with transactions (atomicity), you need to be a bit flexible regarding side-effects. This has been a problem with mnesia, and I have spent a lot of time thinking about how to make database transactions more “Erlang-friendly”, and also more practical in web-scale scenarios. I will talk about some of this at the conference.
As you say, there are many non-trivial problems when designing a database. Most things so far have mostly been fun, actually, but then I’ve been able to draw on much that I’ve done in the past. I’d say the biggest challenge has been the fact that the database really isn’t the main attraction – it’s just a component that, ideally, should have been ready off the shelf from day 1.
Mirko: Feuerlabs is releasing KVDB as part of Exosense. An Open Source Device Programming Framework. Give us an introduction to Exosense and what you are developing. I have read something about it on the company’s blog but I would like to have a presentation in your own words.
Ulf: We are convinced that Erlang is an outstanding environment for developing Connected Device software, and we want to help grow a community of developers who can take a great idea to prototype and product in a very short time.
We see this sort of thing happening now with iPhone and Android apps, which is really a form of device programming. But while the Mobile App development frameworks are extremely complex – much because of the advanced UI features – we envision a much simpler framework, for entrepreneurs in the Embedded Connected Device space.
In fact, Exosense is not so much a framework as it is a collection of useful components – CANOpen stack, Netlink integration, UART, SMS and GPS interfaces, … and of course a nice configuration database.
The Feuerlabs-specific component in Exosense is Exoport, which integrates with our Device Management Server. It allows you to manage your fleet of devices easily, and supports store-and-forward RPC, configuration push, intelligent network link management, etc. We want to make it as easy as possible to hook into our backend service, and this is also where we plan to make money – with a licensing scheme that is extremely approachable for small companies and experimenters, and evolves reasonably as your volumes (and revenue) increase.
Much as Google does – on a much larger scale – we want to help stimulate the emerging market that will end up generating revenue for us. The product space for Connected Embedded Devices is practically unlimited, and we plan to be the go-to guys when you need your devices managed.
A number of the components we’re using are in fact open already, but we haven’t made any noise about them yet. When we do, we will also stand by to follow up with community support.
Mirko: Reading FeuerLabs website I see that you are growing fast. Have you got open positions for developers? Maybe among our readers there are some who want to change their life and join you.
Ulf: Heh, the web site does say that we’re growing (although it doesn’t say “growing fast”), and in relative terms, we grew our operative staff by 33% when we brought Peter LeNoach on board. At this point, we are primarily interested in discussing early adoption, joint ventures and reseller agreements. We fully expect to bring some more people on board later this year, but there is much to be said for being a small company with very sharp people, especially in this phase.
Mirko: Let’s take a deep breath and let’s go back to the past… in the 90s. You spent 6 years of your life in Alaska developing disaster response and military command & control systems. Why were you there and how did you get in touch with Erlang? I know you became the first commercial customer. Were you a simple developer with a great passion or were you in a position to influence the developing of those systems? How did Erlang help you in that field?
Ulf: Yes, I did say earlier that it was, effectively, four years. I went back and forth a bit during a six-year period, 1989-1995. I went there to visit a girlfriend, and ended up building a business together with her father, who was an Air Force Colonel, soon to retire. When I got there in 1989, they had just completed a large military exercise (BRIM FROST), with 50,000 soldiers in the field and 900 exercise controllers, and a home-grown Command & Control System. The exercise was awarded an Overall Outstanding verdict by NORAD, basically since the tasking system worked so well that the work was done before the controllers got there to inspect it. Col John L. Shepherd received a Carnegie-Mellon Golden Nugget Award for his work, and right after the exercise, the system was used to coordinate the Exxon Valdez oil spill cleanup.
There was some controversy there, since the stakeholders in the cleanup wanted to control the information, but reporters soon learned that if they went to the nearest airman with a computer terminal, they could get a much better situation report in real-time than they could at the afternoon press briefing. If memory serves, the solution to this problem was that no information could be entered into the system until after the daily press conference… This was my first insight into the disruptive power of information technology.
The story about how the lucrative maintenance contract we were betting on never came through is best left untold, and the story about how Saddam invading Kuwait lost us a fine deal developing an aircraft logistics tracking system is a good one to tell over beer, but it all led to me having to figure out how to build a distributed Command & Control System with a one-man design team. It’s probably a common trait of young entrepreneurs that they don’t know when to let go of an idea, but it did help me discover Erlang. Faced with such an impossible task, I thought it at least seemed possible in Erlang. Of course, at the time, in 1993, there were no support libraries to speak of (not even ETS tables!), so I had to start rolling my own.
I knew that I needed a distributed database management system. Back then, there were few such beasts on the market (Oracle 7 had just been released, if that counts). I had Sonebreaker’s Readings in Database Systems (1994) and my old textbook by Elmasri/Navate, but I felt a bit isolated in my endeavor. Thinking back, it is striking how much the software industry has changed with the Open Source movement and systems like Github and StackOverflow.
I didn’t get very far with my database that time, but the messaging-based deadlock detection algorithm that I developed then is one that I still plan to use, and am integrating into KVDB now.
Mirko: What do you remember of your 13 years in Ericsson? We read something here and there about Erlang inside Ericsson. Tell us a random thought, something that worth being said.
Ulf: Well, one thing about Ericsson was that it owed much of its success to the AXE system developed during the ’70s and ’80s. Many of their products during the ’90s were derivatives of the AXE. I recall one day as we were ready to release the AXD 301 (which was built in Erlang). I was talking to one of our most senior system architects, Peter Carlsund, and I told him how excited I was – this was the first time I’d had a chance to be part of building such a complex product from scratch and see it all the way to a successful release. He laughed and said: “It’s the first time for me too!”. Puzzled, I told the story to my colleague and mentor, Staffan Blau – also one of the company’s leading architects – and he replied: “It’s the same for me”.
That’s one of the weird things about working for big companies. There is so much legacy, that you seldom get to build something entirely new. And when you do, chances are you won’t get to the finish line (or you’ll be doing something else when the project finally gets there).
Mirko: You have lots of contributions on Erlang Open Source projects such as gproc, jobs, rebar and many others. But as all of us know, time is never enough. If you could have some more time, what is the project you always wanted to start but never got the chance?
Ulf: Hmm… there have been some projects like that. For a while, I thought that if I ever got the time, I would design a social networking system, but of course, there is no need for that anymore. ErlHive was my last brave attempt at making something in that niche, but even then, it really was something I wanted to do years earlier, but didn’t get around to. It certainly would have been a fun project back in 1995.
I also thought it would be cool to build my own DBMS… 
Otherwise, the problem is usually that I try anyway, and don’t have the time to follow up on the project. ‘Jobs’ suffered for quite some time from that, but now I’m actually using it myself (We’re also using gproc, setup, sext, parse_trans and edown, BTW, as well as a number of Tony’s components. We’re eating our own dog food), so chances are much better that it will see some interesting development. I was happy to find that the concept holds up well, though.
Mirko: Erlang has been spreading fast during the last years. Excluding the well known industries where it excels, which kind of industries could benefit from an Erlang adoption in the next years?
Ulf: We are betting on Erlang’s suitability in the Connected Device realm, but perhaps more importantly, I think we will see Erlang-style programming making its way into more and more mainstream programming environments.
Mirko: Let’s talk about serious things. You are a great developer, I know you are a singer and a full time dad. Why not becoming a “writer”? I know that the Erlang Community is waiting for a book written by you (an advanced one). Two questions. Is there a remote possibility to read you in the next… let’s say two years? What kind of book would you like to write?
Ulf: Oh, I don’t really see myself writing a book anytime soon. My current hope is that Feuerlabs will keep me much too busy to contemplate any such project. What I will try to do is to pick up my blogging again, and also write documentation and tutorials for the components we are building here at Feuerlabs.
After that, whether or not I will think about writing an Erlang-related book (or some other book) will probably depend on what holes are left to fill. I recently received Fred Hebert’s excellent book, and I’m sure there will be others.
I have not written so much during 2012. After a little self-retrospective which I have performed in the last days I can say that 2012 has been a tough year for me. But now is 2013 and I have to put in practice all the precious lessons learned, but before I want to put my 2012 in a little post.
