In this edition of the Community Interview series, we sat down (virtually) with Uroš Gruber: life-long technologist and maintainer of the FreeBSD port of the Datadog Agent. Read on for a fun retrospective on geeking out in the 1990’s, developing for FreeBSD in the 2020’s, and some exclusive code commentary and explanations!
The interview
Dan
Hello! Thanks for taking the time out to chat with me. So first off, can you tell me a little bit about yourself?
Uroš
My name is Uroš Gruber. I have my own company, Squarebox, where I provide hosting services for my clients as well as helping other companies with my expertise in this space. I also lead the team of developers at d.labs, where we’re focused on helping clients scale their business from a tech perspective. We also use Datadog with most of our projects now.
My first introduction with technology was with a ZX Spectrum…
Uroš
Oh! And I’m from Slovenia. It helps to mention this because we’re usually very humble and forget about putting our country on the map—in many different areas. We do well enough in sport, but tech is more or less done in the background.
Dan
It's true that Slovenian basketball is quite well known, for example, but less so on the tech side. Let's talk about that for a bit! What's the tech scene like in Slovenia? Is it mostly local, or lots of outsourcing? Are there any sectors that are particularly well-represented?
Uroš
So tech is, in my opinion, good. There are a lot of companies that are doing some great things. Some are more local focused—usually things that are closely related to government—and those are more on the safe side, so not that involved with new tech or major improvements.
Uroš
On the other side, we have many individuals who work remotely for well-known global companies, and we’re known for cryptocurrency tech as well. Of course, some people leave to go to the USA, the UK, or Germany—usually because of the local Slovenian communities there, but also because you can get more financial benefits this way.
Uroš
What I see that lacks a bit is DevOps knowledge. It might be because it’s still relatively new, but on the other hand education is lacking.
Dan
It sounds like somebody needs to organise a local DevOpsDays event at some point (haha). But since you mention education, let’s talk about that.
Uroš
So my first introduction with technology was with a ZX Spectrum that my father bought to play games. He was always a tech person but more on the electricity side, since this was his profession (fixing TVs, radios, cassette players, etc.). I created my first little programs with BASIC. One that I remember the most was typing 2–3 pages of numbers from a computer magazine. It was some game but since I didn’t have any storage media it was so hard to turn it off at the end of the day! I was never into games that much, [but] I really like to create things.
Uroš
In primary school I got my first PC, so I was able to do some more coding. I think it was a 286 16MHz with Turbo, a 40MB hard drive, a 5.25” floppy drive, and Hercules monochrome graphics. In high school I upgraded to my first Pentium. It had quite a lot of RAM, and I needed to sign a government document [to buy it]. I was told it’s because you could guide a rocket or a missile with that amount.
Dan
Your desktop PC was a matter of national security?
The only fun was to understand how computers work.
Uroš
Yes. I was sort of proud of myself. Like the government appreciated me (haha).
Dan
What language(s) did you encounter in this era? Still BASIC I imagine, but anything else? And at school, was it a formal class or more of a club?
Uroš
Yes, I was doing BASIC, everything in DOS. But then Windows 3.1 came and I upgraded to Visual Basic. In school we did some Pascal as well and Assembly code.
Dan
Learning Assembly is a hard challenge, but it often pays off in interesting ways—like when you need to port something to a different OS later in life, for example?
Uroš
The only fun was to understand how computers work. [I spent a lot of time] optimizing code because I thought [that the] computer would have a hard time to do all this.
Uroš
After high school I [went to] university but I never finished. I got in trouble with one of the professors that was teaching “Oberon”, and—wait for it—required that we write code on paper with a pencil! I hated that guy and since he was part of six other classes I thought “well, [I’m] done with this university.”
Uroš
So I started working as a student at a local company, building websites and writing Java and JavaScript. I learned about HTML at university, where I built my first personal home page. I think [the platform] was called Geocities?
Dan
Geocities! That's a blast from the past. All rainbow dividers and under construction animations (haha).
Uroš
It was pure joy. At that time IRC was also popular and we spent countless hours talking about stupid things. I had my own bot that was written in mIRC, though I don’t remember if it was just a script or some specific language.
Uroš
While working I applied to another university and went to week-end classes, but I realised that because of all the work I was doing, I wasn’t learning anything new at class. I’m a self-learner; I really need to understand how things work. [The Internet] allowed me to read so much more about tech and everything.
Sort of a Makefile on steroids…
Dan
Let's talk about FreeBSD for a bit. What got you hooked in the first place? What keeps you with it through to today?
Uroš
I think it was stability and very good docs. I didn’t like Linux because every [distro] had things placed everywhere—like config files were in different folders, libraries were all over the place—in FreeBSD you had /etc or /usr/local/etc, and that’s it.
Uroš
There were also a lot of “wars”, [like] “Redhat is better than Suse”, etc. And having a broken kernel after [an] upgrade was no-go for me. With FreeBSD, things were done at a slower pace, with stability in mind. I once had a server with more than 10 years of uptime!
Uroš
A downside is that the community is not as big as Linux. Also, some software support is not where I want it to be. Actually, that was the reason I started working on FreeBSD port for Datadog. I think it was three years ago for version 4.x or 5.x.
Dan
What is that experience like, porting software to FreeBSD? Specifically the Datadog client to be sure, but just in general? What sorts of questions do you need to ask? What sort of research do you need to do?
Uroš
There is a special framework available. It is well documented. Sort of Makefile on steroids. Some software is very easy to port, [for example] a Python library, as you need just a few lines. Some are more complex. Datadog was definitely on the complex side.
Dan
Complex how?
Uroš
It’s a must to understand what steps and tools you need to build. Then you need to see if [everything you need] is available on FreeBSD. If not, the first thing is to create ports for the missing things. After that you can start running [the] build process. The work actually begins there. [It’s] a lot of trial and error.
Dan
Tell me more about the Ports system and what it means with regards to the Datadog Agent.
Uroš
It’s sort of instructions of how to build software on FreeBSD—commands to execute, how to extract archives, get dependencies, apply patches—as well as rc.d scripts [ed. system-level control scripts], documentation, sample configurations, etc. From there it will install a “pre-build” binary. Without [the Ports system], every FreeBSD user would need to go through all these hurdles, every time, with every software or tool they wanted to use.
Uroš
With Datadog, I found that a lot of paths set in the code are not where FreeBSD keeps things. I needed to patch quite a lot of files. and since it’s Golang, I needed to learn how to read the code and see how things are connected. For every patch I had to run a build, wait, patch, and repeat.
Uroš
What made things a bit more complicated was [that] Python was also part of the system. Combining both the Golang and Python worlds was a challenge for me as well. [It took me] roughly two weeks to finally make it work.
Dan
That's two weeks working a little bit at a time, right? Not full time!
Uroš
Yes, a couple of hours a day.
Dan
What made combining Go and Python more challenging?
Uroš
If you have just one you can use a lot from the [existing] framework. With both, you need to find a way to make them work together; also some custom commands.
Dan
Speaking of code, why don’t we dive in and take a look at some?
I basically had to rewrite a core check…
Uroš
Sure. Let’s start with packaging. I needed a different way to handle permissions on FreeBSD, for example. The default local log path wasn’t ideal due to the filename, and because FreeBSD uses different path [conventions] than Linux.
Uroš
Actually, having to patch filenames and paths comes up a lot! There are adjustments to make in the code, but this also [has to be reflected] in the Makefiles when building at different stages.
Uroš
Speaking of Makefile stages, in the pre-build process, I needed to learn invoke and what commands are actually needed to build the Agent and other binaries, so that I could replicate [those] steps in port. There [were] no ports done in this way [that] use invoke as a tool. This took a while.
Uroš
For the actual build, [there were] some issues with how variable replacement was done, so that it [would] show the correct version name, etc. This was mostly trial and error—and I found out that it needs to use [the] correct case (datadog vs Datadog, for example).
Dan
That’s already a fair amount of work—but it’s just the binary. What about things like configuration files and default integrations?
Uroš
I needed to test [which configuration components] are actually needed for FreeBSD, then remove what does not work at all, and then provide some patch files to not break other things because of these changes. Then I have to make things configurable, [which is] time consuming to check if every configuration works.
Uroš
I set up a separate port that holds integrations. I needed to go through the full list of [core] integrations and check [against the] FreeBSD port repository [to see] what software is supported. After that, [I had to determine the] required dependencies. A few were missing so I created two additional ports for Python libraries.
Uroš
There are also larger patches. I basically had to rewrite a core check because of the different way that FreeBSD handles max open and used open files.
Dan
Wow—that’s quite the effort! I know that the end result is useful for you, but I also hope that you had fun and were able to learn something along the way?
Uroš
A fun part in general was that Datadog switched how to handle Go dependencies, and started using Go modules. I needed to apply those changes and since it was again new for me, [the] job was not easy as it might have seemed.
Uroš
I also pushed some upstream general patches to the Datadog core. All this will be available in a later version and I can then remove those patches from the FreeBSD port.
Dan
Do you have anything planned—or anything you’d like help accomplishing?
Uroš
One big piece that would be very welcome to do is support for FreeBSD jails (sort of like containers), but unfortunately I don’t have enough knowledge to develop this feature. It would be great for others to join and contribute!
Uroš
I would also like to mention [a] couple of contributors that helped as well. First, Mikael Urankar, who made the actual commits to FreeBSD ports tree, since I don’t have permission. Also Dave Cottlehuber, who did the very first port commit! I didn’t have permissions to push to [the] FreeBSD repo and was waiting quite a long time until Dave picked it up and helped with the process.
Dan
Thank you so much for all your hard work on this port, and for being such a good sport with this interview.
Keep the story going!
If you’re interested in developing for the Datadog platform, check out the Developer Tools section of the Datadog docs—don’t miss our regular Community Office Hours either!
