Category Archives: Engineering

RealSelf Celebrates Its Very First Hack Week

Plenty of companies have Hack Weeks but how many include handcuffs? Our guess: none but RealSelf’s, which concluded today.

CTO Matt Woodward explains the ground rules for Hack Week 2015.

RealSelf CTO Matt Woodward explains the ground rules for Hack Week 2015.

In true RealStyle, we kept things interesting by making our very first Hack Week a competition that brought out everyone’s inner developer. At the beginning of December, more than two dozen presenters from all departments took to the stage to make their case and attract co-workers to their team.

A few presenters championed their causes with pun-ridden decks. Others had so many ideas they were standing up more than they were sitting down. One duo even tried for a Tony with their stirring 30-second skit featuring a pseudo-police officer, a reviewer on the run, the theme song from Cops, and yes, handcuffs.

After the truly memorable presentations, audience members (a.k.a. everyone who didn’t present) picked a team. They’ve spent the past 12 days working hard to have a somewhat understandable product ready to show for today’s final presentations. At stake: seeing the idea go into production and, rumor has it, winning the key to the executive bathroom.

So who got the royal flush? The ballots are in but the results, well, those are proprietary. If you want to know the answer, come join our team!

We’re hiring! Apply for open positions now.

Incident on Alaska 609: How to Avoid Getting Called Out by Software Engineers for Bad Design

tarmac_snackDelayed by Design

Our plane arrived late to the gate, and there was some extra mess that needed to be “properly” cleaned up, so we had to wait for boarding. The crowd was mostly tech professionals, coming back from several days in Las Vegas after learning about the latest updates to Amazon’s Cloud platform by day and going out each night. Everyone felt ready to get home to Seattle, and the air had an unusual buzz of conversation.

We boarded, taxied to the end of the runway, and stopped.

Several long minutes later, the pilot explained why: We needed to catch a break in traffic to leave. The plane was heavily loaded with fuel and baggage, one too many free T-shirts in someone’s carry-on.

As the pilot explained, this Boeing 737 was designed differently than others. If we took off with the traffic and an engine shut down, we wouldn’t have enough power to continue our climb and make it over the mountains. We needed a break in traffic, which should take at least another 20 minutes.

People began grumbling. You could tell they were tech people by how quickly they questioned why things were the way they were. “That’s terrible design.” “What was Boeing thinking?”

It’s All About Choices

This crowd had immediately decided that this airplane suffered from poor design. More powerful engines, fewer people on the plane, lighter materials, different wing shapes, many choices could have produced a plane that could take off like the othersone that should have had us halfway home by this point.

The pilot came on again. “I understand there are still questions about why we haven’t been able to take off.” He reiterated his explanation, estimating at least another half hour until there would be a break in the traffic. He asked the flight attendants to serve water and snacks.

Even after cookies were distributed, people were not placated. Questions persisted.

“Couldn’t they have designed it differently?”
“Why does the airline fly a plane with this deficient design?”

I struck up a conversation with one of the guys questioning the design. “This is the airplane they fly to Hawaii,” I told him. “It’s designed to fly long distances over water and return to land even if one engine stops. According to the pilots, we’re loaded heavily today. Under different weather conditions, we might be taking off with the traffic and none of us would be the wiser.”

As we talked, he began to understand why these choices were made, why we were in this situation, that it wasn’t all foolishness. This design was well-suited for long and over-water trips, and as a consequence, it didn’t have the right performance in this situation.

Define Your Purpose

Every day, software engineers are faced with choices that define not only the way they do their jobs but the resulting systems they’re building. They have to constantly ask their own questions.

What are we building this software to do? How are we building it? Should it be fast? Should it behave perfectly in all situations, like software controlling medical equipment must? Should it be delivered quickly? Engineered for future flexibility? Should it be complete, or should we build the minimum viable product required to get user feedback?

Everyone has their bias. The things engineers value tend to color their work. Prior to writing the first line of code, the team building the software needs to agree on why they’re building it, what the business needs, and what that means about how it should be built.

With our own personal points of view, it can be difficult to have the discipline to hear, understand and align the solution with the user problem and business context.

To grow our company quickly, we need to focus our resources on the right problem and solve it as quickly and correctly as possible, so we can get on to the next thing. The system design should reflect this. What we build might not be perfect for all situations. It just has to be right for our purpose.

Explain Yourself

In a world where a highly trained pilot can’t effectively communicate with a planeload of software professionals, communication between engineers and business leaders can just as easily fail, derailed by jargon and a lack of common context.

Engineering leaders need to keep reminding their teams of the top business priorities, focusing resources on solving the most important problems in a way that keeps the company growing. And everyone has to remember that the way we explain the reasons for our choices is nearly as important as the choices themselves.

As RealSelf’s Chief Technology Officer, Matt Woodward supports enterprises with systems ranging from mobile applications that fit in your hand to e-commerce search and transactional systems spanning data centers across the globe. After hours, you’ll find Matt at a hockey game, snowboarding or, on the rare clear Seattle night, observing the universe through a telescope.

If you’re interested in talking with Matt about opportunities on RealSelf’s technology teams, learn more about our open roles and get in touch.

Engineer Turned Rap Star Has His “Code Goin’ Up”

Being the star of a rap video was never in the cards for me. I’m a software developer, gameCodeTuesday_BTS-8r, and cellist born to strict, conservative Asian parents. So when Brandon O’Larey, our executive video producer, approached me with the idea for “#CodeTuesday,” I’m sure you can believe I was stunned. So much so that, at first, I declined.

It wasn’t until we met for coffee the next day that I got a true sense for his determination and dedication to the project. His excitement was infectious. He told me he believed I could do it. Who could say “no” to that? (Our CEO, Tom Seery, couldn’t.)

CodeTuesday_BTS-7

It took the collective effort of the engineering, product, and design teams to land the lyrics for the rap. We went line by line, a group of nerdy RealSelfers pulling together to write Grammy-worthy lyrics which would inspire software developers far and wide. You know you’re in good company when a colleague suggests you add the words “tight-ass code” to improve the flow of a line. There were no merge conflicts with that change.

I practiced and practiced until I could see myself on stage with Drake at CenturyLink Field. It wasn’t until the morning before tCodeTuesday_BTS-3he recording session that I began to feel anxious. Lucky for me, Brandon had a 24 of Olde English and a brief hip hop history lesson prepared as we walked into the studio. After a few short hours of head bobbin’, hand wavin’, and beat droppin’, we were out the door brimming with confidence. This is going to be good.

The track was ready and the beat was right, so we threw a party in the office. If you cover the windows, turn down the lights, and blast some rap music, our office becomes a club and our team hot and poppin’. We didn’t wait for an earthquake to bring the house down. I’m glad we caught it all on camera.

CodeTuesday_BTS_BW-4

Want to join the RealSelf Crew? Apply now!

Let’s Talk About Debt

photo courtesy of flickr user f-r-a-n-k

When we’re talking about what we’ve done, are going to do or haven’t done, one phrase inevitably comes up: “technical debt.” Scott, our VP of Product usually corrects the concerned engineer: “that’s technical leverage.

Indeed, as a smaller, bootstrapped company, we must carefully choose our activities. We need to focus our efforts on impact — proving our ideas cheaply, intentionally choosing to avoid work until we learn from our users what the next step should be.

Fortunately, unlike financial debt, there are clear decisions to be made about technical debt. You don’t have to pay it all off today…you may not ever need to pay it off.

The way you achieve comfort with your technical, ahem, leverage is to ditch the euphemism and begin to discuss the real issues.

We started a conversation about technical debt, what it really is and how to deal with it. The first thing I tried to do to get my head around the problem was to think about what it actually meant. The phrase technical debt ends up meaning nothing, or encompasses so many things that it becomes a trigger for more work, big schedules and general pessimism.

Technical debt can be benign, like a quick hack to reduce the work required to get a release out as easily as it can be system instability waiting to cause a 3AM site outage. As I continued to list of all the things, Anthony, our VP of engineering, cut to the core of the issue: risk.

Whether debt will have an impact on us and how likely we are to feel that impact is what we really need to be thinking about. With that insight, Anthony adapted the binary risk analysis model to help us understand and categorize our debt. The process is ongoing, but we’ve started building a model to help us get comfortable with the debt that shouldn’t worry us and get proactive about the debt that should.

Optimizing Slim Routing

PHP image courtesy of flickr user stethoscopeis a general purpose programming language–but it acts a little differently than most (Ruby, Java, etc.) when used as a plugin to your web server. A Ruby on Rails app starts up and loads a bunch of things into memory and continues to run, but an Apache+PHP application is constantly dumping your PHP code and re-starting everything for every web request. Thus, while you can have a lengthy startup time in Rails and it’s totally cool, you must optimize your PHP code to do as little as possible all the time to keep it fast. This means keeping careful track of how your code loads up all of its dependencies–and not all PHP web frameworks do this elegantly.

Here at RealSelf, we do continuous integration and rapid deployment, and we iterate on things quickly. Sometimes code that starts out as an experiment turns into a key component of our infrastructure–even if its architecture wasn’t initially optimized for performance! So when we chose Slim to iterate quickly on a new API service, we got something up really quickly, and then slowly watched it grow in importance and response times. Recently, this situation became untenable as server response times grew to >300ms per request. We are getting between 1-3000 requests per minute for this particular service and end-users were starting to see the effects. Luckily, there was a simple solution to get around Slim’s deficiencies.

We discovered that each single route that we registered added a significant overhead to each request. As an experiment, we doubled the number of routes registered, and saw that we roughly doubled the response time of the server. Ouch! This service has a rather large API, so the number of registrations is significant here. Since this is a PHP app served by Apache, we quickly realized that we really don’t need 99% of these routes registered on a given request: Apache already knows which route was requested when it starts executing the PHP code, and PHP exposes this information in the $_SERVER['REQUEST_URI'] bucket. So, we really only need to register ONE route with the Slim framework. Nice.

We parsed out the REQUEST_URI and made a little registration routine that selects only the specific section of code concerned with registering that route, and the results were dramatic:

We shaved ~250 milliseconds off of our request time, bringing most requests under the ~100ms line. Notably, PHP is _still_ taking about 85% of the total request time, so clearly there are more optimizations to do. We should be able to get our response below 50ms. Really, all PHP needs to do is load some extremely targeted business logic (usually not even any logic, just a database query) and then turn that into a JSON payload to hand back to Apache. If we’re doing anything else, it’d better be for a really good reason!

It’s worth noting, however, that in other languages this wouldn’t be an issue at all. A Java web server would load all of this overhead up on app startup and code would be loaded, unloaded, and garbage-collected whenever the JVM sees the need. PHP is a very different beast, and needs to be treated very differently.