Adzerk Team Blog

It seems to come up fairly often that members of the UX community are hesitant to share their experiences because the principles of UX are perceived as too abstract to be easy to teach to others. I have to disagree. Teaching UX is no different than teaching any other subject. The reason I’ve been thinking about this lately is that I recently worked with Tutsplus Premium to create a 17 lesson screencast series on the basics of UX Research and Design.

Many exciting adventures were had in the recording of the screencast itself- for one, I didn’t have access to a recording studio, and never realized how loud my life is. Which is how I came to be recording screencasts on the floor of a closet. With my cat sitting next to me. (To keep her from pawing at the closet door and alerting her louder, less well-behaved younger brother to my location.) But that’s a story for another time.

She thought it was very exciting.

This originally came about when Envato decided to relaunch their Tuts+ Premium site with an emphasis on screencast content. I had written for them in the past, so they approached me with the suggestion that I do a course on UX. They had little to no idea what content such a course might contain, so they left the curriculum completely up to me. Luckily I knew exactly what kind of content I wanted to create. I wanted to make the complete guide to practicing UX. The one I’d wished I’d had when I first graduated.

UX is a fairly young field, and there’s a lot of disagreement even among UX practitioners about what exactly it is that they should be doing- conducting tests and surveys? Organizing information? Writing micro-copy? Tracking demographics and usage data? Making graphics? Writing code? All of the above?! I have to admit that I am one of those who further muddies the waters by refusing to say, ‘I do this, but I don’t do that.’ This is because I have always worked on small teams, where a willingness to wear many hats was crucial.

The fact that the field covers such a wide array of topics, each with their own opportunities for specialization within makes it a unique challenge to teach at an overview level. Someone broached the question to me ‘How do you approach the difficulties in teaching UX Design? I find it incredibly difficult to do that without a project handy and, even when one is, it’s a test project that’s so devoid from reality as to be trivial.’

Let’s make a comparison, just for argument’s sake. I think we could all agree that the field of graphic design is also vast, broad, and contains within itself a variety of areas where specialization is possible. And yet, the internet is overflowing with graphics tutorials. Some of them deal with underlying principles, some are very small and specific. I see tutorials all the time for creating one effect or another in say, Adobe Photoshop, and yet, we don’t seem to hear a lot of hand-wringing that no one is teaching the overall principles of design, or that lessons in graphic design are too trivial to be useful or relevant. Many small lessons add up to a body of working knowledge. It’s valid to apply that approach to UX as well.

It’s ok to create a lesson on something that’s small and specific. Even a trivial example goes to demonstrate principles and techniques. Besides, you can’t teach everything at once. So break it into pieces.

I broke down my experiences with the lifecycle of UX into three parts:

• Research
• Design
• Implementation

In the research section, I covered topics like:

• Defining the anatomy of the user experience,
• Finding out who your users are,
• Identifying and prioritizing usability problems,
• Designing tests and working with users

In design, I covered topics like:

• Creating an information architecture
• Designing interactions
• Creating wireframes and UX considerations for visual designs

In implementation, I talked mostly about things like tools and resources, but also about

• getting organizational buy-in,
• project workflow,
• project management methodologies,
• and how to win arguments with data.

In each topic, I used an example from real life. In some cases, I used the same example from lesson to lesson, building on it the way that I would have in real life. In other cases, I took an existing website in the world, and demonstrated how to apply analysis techniques to the content that was there. I laid a strong emphasis on the idea that I was showing one way to do it, not The Right Way To Do It™.

I think that’s an important distinction to make in many subject areas, but especially in an area like UX, where the best outcome is so heavily dependent on your particular context. But I do disagree with the notion that demonstrating how to design a navigation menu on a web application is not also demonstrating principles that would apply equally well to say, a mobile app.

The overview was far from comprehensive, but it gave at least a snapshot-level introduction to each topic included, and feedback has ranged from the positive to the outright ecstatic. You don’t have to be a regular on the national conference beat to make positive contributions to the community, just be willing to share what you do every day. There are a lot of people who are eager to learn from you!

To the aspiring manager nothing throws a chill more than seeing idle people at work. There they are - the shiftless dregs - doing nothing: staring out the window at the Korean taco truck, reading an article on Smashing Magazine, checking the fantasy basketball standings. Surely there is something they should be doing. Something to add to shareholder value. Aren’t we paying these people? Why are they loafing about? Why aren’t they coding up some APIs?

Managers are rated on how they manage people. “Good” managers get people to produce, while “bad” managers can’t get anyone to do anything. Often managers are held to a single overriding standard - did your people make lots of stuff or not?

Someone who is judged by how many web pages, bug fixes, lines of code, double stuffed burritos, cinnamon decaf soy chai lattes, or cars the people under them produce will do just about anything to make sure that metric is met or exceeded. No one wants to appear unreliable to their boss.  (We’ll discuss why performance reviews are a terrible idea some other time.) 

If more is better, then every iota of energy should be devoted to the work. Idle time is the enemy. Idle employees means you, the manager, are not doing your job. The worst sin is allowing anyone to be idle with nothing to do. The horror!

The frightened manager has a mighty selection of tools to pick to combat fiendish idleness. The best tool they have is the schedule. This is the hammer that will drive every nail while crucifying the poor son of a bitch who can’t pull his own weight and crank out shareholder value. Those people estimated the work, they committed to it, and BY GOD, IT IS WRITTEN ON THE SCHEDULE.

The schedule is the purging ray of sunlight into the abyss of the employee morass. With enough effort and sufficient concentration every moment of every able bodied hand is directed to productive channels. From the moment they walk into the office, the schedule ensures that they will be busy. There will be no wasted time. Victory, at last!

A Victory, you say?

Thinking as above (and the performance appraisal system that reinforces it) paradoxically degrades output over time, it does not increase it. In fact pushing systems to capacity creates delays. Creating “slack” in the system (and in every case when we say “system” we actually mean “real people”) is in fact a productive measure, if not a humane one. 

The biggest factor in how people adapt to a workload where they must engage/disengage/re-engage their mental faculties throughout the day is context switching. A context switch means that someone must deliberately reorder their mental map of their current work to re-map to a different model of work. This remapping costs time - not just the time it takes for the moment of the switch, but also including the time lost as a person must re-adapt to the new model. Human context switching is evil. Manager should seek ways to help employees eliminate jumping from one job to the next like a drop of water on a hot skillet.

To a naive manager the directive to create idle time is a shocking and painful epiphany. But it’s critical to understand that idle time is important. Schedule slack increases chances that the right person is ready at the right time for the right job. Slack also provides a buffer against piling multiple priorities on a single person, which only makes the person complete all work later.

It’s also important to understand that the purpose of your endeavor is not to make sure everyone is busy. The goal is to make money - to produce value for which your customers are willing to pay. Ask yourself - is it more important to have 1 task 100% done, or 9 tasks 90% done? Which is going to bring real value to your customers?

Once you’ve shaken off the illusion of 100% utilization, reexamine your tools. Do you maintain task schedules? Ask yourself what the real purpose of the schedule. Do you use it to keep everyone busy, all the time, because having idle people is bad? 

Unfortunately, even if a manager realizes that the very process of managing people perpetuates problems, they are probably trapped. Unless the top layer of management understands the same principles of capacity and slack they are unlikely to support substantive changes. But real changes rarely come without pain. 

The past week has fucking sucked.

Let me back up slightly and explain what’s been going on at Adzerk. In the past 3 months we’ve turned over our support department, twice; which means we spend cycles interviewing numerous candidates. At the same time, we’ve increased our customer base by 300% and due to the increased impressions now running on our platform our engineers are helping Adzerk through some… growing pains. Not to mention the partnerships we’ve recently announced which bring their own integration issues. 

Any one or two of these issues alone would be manageable and routine but all of them together and then to be short handed really make life interesting.

But you work for a startup, you have flexible hours and no dress code and hang out playing nintendo all day. Life must be grand! — Fuck off. 

We tackled the interviewing like it was a top priority. I believe when you need to bring in headcount you should attack it with laser focus to make sure you hire the right candidate(s) and you hire them fast. Putting it off is only going to prolong the issue. After 2 straight weeks of interviews we found ourselves with great candidates to choose from and came out with 3 new hires we couldn’t pass up when we were only looking for 2. Great, no more interviewing, lets put out some fires. 

Due to the turnover in support reps and increased customer base we were suffocating under a mountain of support tickets. It was all hands on deck as I managed the support queue and delegated like a mad man. We had everyone from engineers to the CEO handling support cases. 12 hours a day of support cases doesn’t leave time for blogging or marketing or giving attention to any number of other tasks. 

There is a silver lining though: when everyone answers support cases it means everyone feels our customer’s pain. We all have a better appreciation for areas we need to improve in our product and support processes.  I’m really proud of the team and attention we gave our customers over the past few weeks. We came out of this a stronger team, stronger company and stronger product. 

This past week we finished rebuilding our reporting infrastructure so scale issues have been slayed. Our engineers now have time for bug fixes and feature work (when they aren’t answering support tickets). We’re smoothing out the issues with our integrations and our partnerships are bearing fruit as more customers adopt our platform to take advantage of these integrations. 

On a personal level, I spent this week still handling the support queue instead of my regular VP of Product Marketing duties. But I also got to spend time bringing our new hires up to speed on the product, the industry, and our customers. This is going to be a great support team.

I’ve dealt with tough weeks at past jobs, jobs that I’ve been committed to. Startup life is different though, the level of passionate commitment and responsibility as an executive is so heightened that when shit hits the fan it can knock the wind out of you - and you can’t go home and escape it.  Make no mistake though, I wouldn’t trade these weeks for anything, they’ve been some of the most valuable of my career.

Working at a startup isn’t all fun and games, but that seems to be all anyone wants to talk about —  just don’t forget the occasional desire to jump out the nearest window.

A few Fridays ago, I was at my favorite bar in downtown Raleigh with some friends. We were hanging out and having a casual conversation about startup culture and how it compares to corporate culture. A young guy with a button down shirt and a loosened tie was ordering drinks at the bar and overheard our discussion and introduced himself. His name was Jonathan. He passed his ordered drinks over to his friends and decided to stay and chat with us for a few more moments. Jonathan is 25 and works at a very large local corporation in RTP as a software engineer. He was hired right out of college to write internal code for his company, and he’s doing very well for himself. It came in passing discussion an half hour later that he’s making $80,000+/year (that’s a lot for an entry position here), has a mortgage, a fiancée, and a dog. From the outside looking in, it seems that Jonathan has everything working out for him. I later asked him directly, “…but do you love your job?” He paused, looked down to his drink, and replied, “I love my team, but I f***ing hate my job.” 

Now, this blog post really isn’t meant for everyone. You could be very offended because you like your job, or it’s possible that you can really relate to it. I’ve got friends that work at large companies that absolutely love their job. This isn’t for them. Just ignore this post. This is for the guy who browses Hacker News at his cubicle during his break under soul-sucking fluorescent lights. The guy that is sitting back watching the tech world move by while he’s writing code that he hates and spending his days making sure the TPS reports are filed correctly. You need to break free, dude. Now is the time to do what you want.

Your Job is Important - Let’s face it. Your job is a huge part of your life, though it shouldn’t be confused with your job being your life. If you spend an hour in the morning getting ready for work, a half hour commuting there, a half hour commuting back, and you work 8 hours in one day, then that’s 10 hours of your workday that is devoted to your job. Assuming you get 8 hours of sleep, that leaves you with 6 hours in a day to actually live your life on weekdays. When you wake up, do you look forward to going to work? Are you ready to tackle today’s problems? If the answer is no majority of the time, then it’s time to move on. You know people that work in startups. You’re also aware that in some weeks they work 60+ hours. They might be pulling their hair out some days, but I bet if you chat with them they wouldn’t trade their job to work in a cubicle farm on depressing software.

Build Something You’re Proud of - You know that you would rather be creating something. Instead, middle managers want you to make sure the internal reporting software is correctly calculating bullshit figures for their bosses. The Java Struts web app that you’re working on needs to be able to support IE 6, especially since you’re not able to install Firefox or Chrome on your locked down desktop. 

You were like me when you were a kid. The action figures sat in the corner, but you were on the floor with your legos. You built things. It’s what you do. You’re a creator. Why not build something you’re proud of? Don’t you want to be one of the developers of an app that is on all of your friends’ smartphones? Don’t you want to “wire in” one day and get lost in your project? 

“Get action. Do things; be sane; don’t fritter away your time; create, act, take a place wherever you are and be somebody; get action.” - Teddy Roosevelt

The Time is Right - Some say we’re in a bubble, and we’ll all be without jobs soon. They say that we’re heading for a Web 2.0 dot com bust like we had last decade. That topic can be debated for hours alone, but one thing is for sure: we’re in unprecedented times. There has never been a time in history where a few people with laptops in coffee shops can create an online empire. Never before have people been able to create wildly profitable and highly valuable companies with little funds.

Job Security is a Myth - I’ve only been programming professionally for a few years, but I’ve learned that all jobs are temporary. That cushy job you have that pays for your nice car and mortgage can easily disappear with budget cuts. Those certificates that you’ve been working on for internal promotions probably won’t be valuable at the next company you apply to. You could be laid off with the 25% of your company even if it gets acquired. 

Shit happens. Jobs come and go. Why not let shit happen while you’re doing something you love? 

But I Like My Paycheck - Don’t let society pressure you into thinking that a corporate job is better than a startup job. When I studied CS at UNC, it was kind of assumed that you would go write software for some large company. I recall looking at a slide that compared salaries of different job sectors in the tech field. All the web jobs were at the bottom of the list. Sure, that may have been true 5-10 years ago, but having a job at a smaller company can be very rewarding financially and mentally. The problems at some web companies are just as difficult or harder than the ones people solve in office parks. There’s also a chance (though much smaller) that you’ll be able to retire on the equity you earn.

Everyone is Hiring - Seriously. At least it feels that way. Even in the Triangle. Money is flowing into tech. I would love to see an unemployment statistic of coders vs. every other occupation. I’m willing to wager that it’s under 3%. If you’re unemployed, not making rent, but can code, then you’re doing something wrong. The Wall Street Journal just name “Software Engineer” as the best job of 2012. They measured “physical demands, work environment, income, stress, and hiring outlook.” If it’s the best job in America, it’s easier than ever to find another job, but you hate your job, why are you still there? 

Do yourself a favor. Go hack on something this weekend. Create a pull request on Github. If you’re in a city that does Startup Weekend, go participate. There’s one this weekend in Raleigh, NC. Life’s too short to be creating shitty software.

“In the end it’s not the years in your life that count; it’s the life in your years.” - Abraham Lincoln

Kacy is a software engineer at Adzerk. He believes he has the best job in Durham, NC.

I would like to hear your thoughts. Please feel free to post them in the Hacker News thread or in the comments below.

I have written before about Adzerk and our choice to work in the American Underground. I also wrote a post last year about the cancellation of the Launchbox Accelerator program and the negative effect it had on the community and the Underground. 

I have good news. Chris Heivly has started a new accelerator Triangle Startup Factory and they are back in the exact same spot they were in before and a new class of startups have already started.

It has been about 2 weeks and I can’t stress enough how incredible it has been to see the increase in activity in the AU, not just from the 5 new startups that are part of the class but the people coming down to see them, talk to them, and work with them. In just two weeks I have already been down to their office for a launch party, a lunch session where I got to discuss a couple of things I have learned from building Adzerk, numerous hallway conversations about their businesses, a rousing game of MLB 2012 with one of the co-founders of Berst, and the nice founders at Ruzuku bought me a beer to discuss different business models.

Not only do I get to interact with the new companies and founders, but I also run into people who I haven’t seen in awhile who are coming down to talk to the class. The AU is one again the place where on any given day you can run into a plethora of founders, investors, and advisors.

I only realized after the last couple of weeks that I didn’t really understand just how exciting it would be to have an accelerator back up and running in the American Underground - by the time Adzerk moved in (Jan 2011) the last Launchbox class was technically over and it was just 1 or 2 teams left working in the space. Now that I get to see an accelerator running at 100% it is truly impressive.

It’s never been a better time to be in the AU - if you are looking for office space I believe there is one spot left. The first beer is on me.

-James

Feature flags are an indispensable tool for working with continuous integration and deployment. They allow us to add new code to the app without breaking things, keeping everything safely hidden until the big reveal when we turn them on and our customers get to see the new feature in all it’s shiny glory.

When we first started with our continuous integration system, we implemented feature flags to allow us to do development and commit our code without instantly deploying unfinished features to production.

We soon realized that the product management and customer support teams needed an interface to be able to independently turn on and off feature flags without requiring them to ask a developer to get in the database every time. Hence, this screen was born:


The feature flag management tool.

It was awesome! For awhile. Then this started to happen:


A large and growing feature flag collection.

After all, in the interests of maintaining sanity in the codebase, we can (and should) create as many feature flags as we need to keep separate pieces of features separated.

Nonetheless, we started to realize that a problem was brewing. In the developer’s minds, when a feature was cleared out of AgileZen, as far as we were concerned, it was done, and we didn’t think much more about it. But often the feature flag was not enabled for many customers, as the product team worked to make sure everyone was comfortable with the changes. In fact, some old feature flags were not enabled for everyone for months after deployment.

The initial idea was that we would remove any given feature flag immediately after acceptance testing, ideally before the Friday of the week it was deployed. Quick, smooth, boom. Features to customers. What actually happened was that sometimes features require debugging, or customers were unsettled by their page layouts flying around on them. So we started taking a more cautious approach, beta testing for early adopters and volunteers, and then, when features are rock solid and we’ve had ample time to notify everyone what was coming and exactly what it would look like, then and only then would we remove the feature flag from the build.

What was happening in practice is that a feature flag would get turned on for a few people who had requested the feature, and then, as often as not, everyone would forget, developers and product owners alike. And new features would languish in purgatory for weeks at a time. Of course this wasn’t always the case, but it happened enough to make it a problem.

We track many things in our office visually, to help with the short term memory loss we all suffer from as a side effect of our respective Reddit addictions, as well as to provide transparency to the entire team.

James came up with the idea of making a feature flag map on the wall, similar to the application sticky note map that I’d made a few months earlier:



A sticky note map of the adOS UI.

The process works thusly: when a new feature flag is created, a note card is also created, dated, and added to the wall. This way, at a quick glance everyone can see which feature flags are around and how long they’ve been alive. When a feature flag is removed from the build, it’s note card is removed from the map. This gives a highly visible view into the speed at which we build and deploy new features to *all* of our customers, and helps remind us when an old feature flag is languishing.

The result looks like this:


Sanity and transparency!

While we’ve only been using the system for a couple of weeks, it’s already helped us be more aware of the feature flag issue, and to keep the build clear of old feature flags.

Do you use feature flags in your app? How do you handle keeping track of them?

I was planning on writing another blog post until the last minute when I decided to write this one instead due to some hard times that have hit home in the triangle area.  As I am sure everyone has heard, iContact was acquired and then scaled down causing some layoffs to old colleagues of mine who I also consider friends.  This got me thinking about what would make this situation easier on everyone involved.  Of course a new job would be great, but there are ways to get jobs a lot faster when unexpected things like this occur.  The one that I have always done and will continue to do is network with different people!  If you keep your well full prior to getting thirsty, chances of going without water are a lot lower.

When it comes to keeping food on the table in this economy and during these tough times, it is all about who you know.  That is why it is extremely important to build up a network of connections that you can fall back on if needed at any point.  You may be in a situation where you think you are safe now and love your job, but you never know what life brings so my advice is to always be prepared.

The best way to network and build both professional and personal relationships is by going into every conversation as a sales opportunity.  And when I say this, I don’t mean telling your whole life story and why you are the best at what you do.  Make it a goal that when you are done with that conversation, they leave with the feeling that you are a stand up person and it is someone you would want to get beers with and hangout with again.

Another way to generate a great network is to help people in need that you may or may not know.  This act alone could open doors that you may have never expected.  If you put yourself out there for someone not expecting anything in return, chances are they are on your side for the long haul.  If they don’t reciprocate in the future, don’t count it as a loss because I guarantee you will feel pretty damn good about yourself.  I will take karma on my side any day.

These two things are great, but if there is no follow up, you may as well not even start the process.  There are tools that make it a lot easier to follow up today so why not take advantage of them?  One of the best tools out there for professional networks is LinkedIn.  After each personal or professional interaction, it is smart to connect on LinkedIn to keep that well full.  Then, if you come across an article or something that will help one of your connections, send it off to them and stay in touch.  People remember those little things and it can make a difference at one point down the line because you never know who your connections know and so forth.

It also never hurts to go to various networking events in your area.  This is a great way to build up more connections locally while you are still in a good situation.  You may meet someone there that is not in a good situation and have the opportunity to lend a helping hand.

Of course, I never wish these hard times on anyone, but I also think it is a life lesson to learn from at the same time.  It is extremely important to realize that all jobs are temporary as you never know what may happen.  So, if you are consistently networking you will put yourself in better shape in the future.  Lastly, if there is something I can be helpful with for anyone, don’t hesitate to reach out.  I may not be able to help personally, but I can definitely point you in the right direction.

-Ron

In high school I was terrified to ask out the girls that I liked - which meant I didn’t ask very many of them out - which meant I didn’t have very many dates or girlfriends in high school. It seemed like a wise strategy at the time - I didn’t want a girl to reject me so the best way to avoid that is to never ask a girl out.

I believe embracing rejection is one of the keys to leading a successful life.

When you look at the lives of successful people you find people who are not afraid of rejection - in fact they not only aren’t afraid of it but they embrace the rejection in their lives. Instead of letting rejection bring them down and discourage them from trying again they use that rejection to motivate them and to help them learn. 

They understand it is just part of the process (of dating, selling, raising money, life) and don’t take it personally.

When I ran ad networks one of the main things that held me back from making those ad networks even more successful was that I wasn’t all that great at ad sales. I had no problem talking to people, taking orders, offering suggestions and packages - but when it came to reaching out to new potential advertisers I was pretty bad. I would put it off and finally send emails on Friday nights almost subconsciously trying not to get a response. It all came down to the fear that the potential advertiser was going to say no - and how could I avoid that potential no. I would do the same thing when it came to getting renewals - I would delay asking about the renewal because of the fear that the advertiser wouldn’t renew.

My fear of rejection slowed down the growth of my business.

When I first set out to raise money for Adzerk I talked to a small handful of investors - assuming that one of them would surely be interested. We had some good conversations and good interest - but all of the initial investors in that group declined to invest. (either like a gentlemen with a gentle “no” or the more traditional “never respond to your emails” rejection). Instead of quitting, or taking the rejection personally, we kept pushing forward and found the right group of investors to fund our seed round.

I didn’t let rejection stop me - but I did let it slow me down. We talked to investors slowly - adding them in a very linear process that meant it took almost 6 months from first conversations to closing the round.

As I have set out to raise our A round I have taken a very different approach -  to treat raising money as more of a sales process. My goal is to talk to 60 investors with the idea that somewhere in that list is the perfect firm or pair of firms to lead our round. I am embracing the idea that most of them are going to reject me - for a hundred different reasons. That is the point though - just like dating to find the right girl - you are just asking out investors and trying to get a match.

It’s not about you

The key to embracing rejection is to realize that it’s not about you. When a VC rejects you and your business - they are rejecting you for one or more reasons that most likely totally outside of your control. Those reasons that a VC rejects you (1st time entrepreneur, crowded space, wrong industry) might turn out to be the exact reasons that another VC decides to lead your round. You can’t look at rejection as a personal damnation of you or your business - it’s just a part of the natural process of finding the right investor/company partnership. 

This applies to all the rejections you get through life - when looking for a job someone doesn’t hire you because you aren’t a good fit. That doesn’t mean you are at fault in some way - it means you need to keep interviewing and submitting your resume until you find the right fit.

Rejection is just part of the process - only when you take the personal emotion out of it can you embrace it and use it to your advantage.

P.S. I am happily married with two kids now - so it all turned out fine in the end.

-James

No snow is there, nor heavy storm, nor ever rain, but ever does Ocean send up blasts of the shrill-blowing West Wind that they may give cooling to men. 

Any cloud management service worth a damn does 1 thing and does it well: provide leverage. Everything else is frosting on your cupcake - it’s sugary fun but not enough to keep you alive.

What do I mean by leverage? I mean that the service allows you to amplify your energy through time and space. The service must give you a lever to move the world with the least amount of effort.

This is critically important when we consider the opportunities and challenges operating a business infrastructure. It’s easy to get lost in the volume of decisions and data you have to manage. Consider that the “cloud” gives us relatively inexpensive scaling. We can throw vast amounts of ad hoc resources at a problem until it’s no longer a problem. (I call this the Zapp Brannigan Strategy.) 

Unfortunately, as you begin to expand the number of nodes in your business, the amount of information you need to process explodes. Your biggest moment-to-moment attention sink will be handling all those cloud entities.

You need to decide what’s important to keep an eye on. Are key processes running? What about disk space? How many requests per second is that nginx server handling? Is that message queue filling up?

At Adzerk, we use Chef as our infrastructure-as-code tool. We can easily define the “what” for each machine without a heavy investment in the “how”. The advantage in using a tool like this is again, leverage. We can define a policy for how an operation will start, operate, and report data once without ever needing to re-create those steps when new resources join the operation. Our attention is freed from that task for other activities (like Terminator2 pinball).

While an infrastructure tool is fantastic, it only solves part of the problem. There’s more information available than you can simultaneously ingest. You need to filter out the distractions or at the very least prioritize the data. You need to know which operations are functioning, which operations are failing, and which operations need to change.

We’ve used a couple of cloud management vendors. Here’s what we’ve learned:

Cloudkick

“From the inception of Cloudkick, we’ve followed one paramount goal: make the lives of system administrator easier.”

The single greatest advantage to using Cloudkick was their Chef integration. We didn’t have to define explicit alarm or metric policies for nodes in our business operations. Instead we defined a single policy that essentially said “If a node has this Chef role (management application server), report these things (disk usage, cpu usage, Nginx requests/sec, etc)”. 

Cloudkick also included some visualization tools, a plugin architecture, and a fairly compact dashboard layout. These features were tasty but not nutritious. 

Unfortunately we stopped using Cloudkick after a month or so. We found their cusotmer service to be frustrating and unresponsive. For example, we had an issue with debugging metric reporting for a specific plugin and didn’t hear back for 6 days. I don’t mean we didn’t get a resolution for 6 days. We didn’t get ANY response for 6 days. There was no “hey, we got your message, and we’re working on it”. Nothing. 

We’re not sure why this happened. We think it’s because they were acquired by Rackspace. Large corporate acquisitions of small startups are the tar pits of innovation - where hungry young entrepreneurs go to die.

Pro-tip: when you put up a landing page, don’t say: “Cloudkick is going away in around a year or two, but not before we release something ten times as awesome.” This doesn’t inspire conversions. It encourages prospective customers to quickly find an alternative.

Server Density

“Easy Server Monitoring”

We needed a quick replacement for Cloudkick and Server Density appeared to be a welcome contender. 

Unlike Cloudkick, it felt as if the technical staff responded to our queries for help and guidance. We didn’t feel like our problems were very deep but Server Density still addressed each question within a day or less. When we had a specific problem with a plugin or API call, the technical assistance was never flippant or dismissive. It felt like they wanted our business, which was a welcome change.

Of course, Server Density doesn’t integrate with Chef nearly as well as Cloudkick did. Creating a Chef recipe to install the agent software wasn’t difficult. However Server Density doesn’t apply monitoring or alarms based on metadata associated with our nodes. That must be done manually, which is a huge loss of leverage. Now, when a new node is created, you must manually add the alarms or copy the configuration from an existing node. In either case it’s not “automagical”. And for large numbers of nodes it’s unwieldy. 

We certainly hope that Server Density evolves quickly enough to address this concern. We don’t want to change cloud management and monitoring providers again, but if a sharper, hungrier option is available, we’ll consider it.

And that again highlights the advantage of leverage over the nodes we already manage. With our infrastructure tools, deploying a new monitoring agent is rather trivial. We can swap vendors as easily has we can edit a JSON file.

At Adzerk, we practice continuous integration. This means that any new code that’s committed to github will immediately run the gamut of functional and unit tests, and if it passes, be promoted to production, all in a matter of minutes.

This is amazingly convenient and easy, especially for the backend developers, who simply need to place a new or incomplete feature inside a feature flag to render it safely hidden and quarantined from the rest of the application.

Things are a little less simple when it comes to CSS.

Adzerk’s CSS used to be a single monolithic file of nearly 13,000 lines. Prior to the implementation of continuous integration, it was time-consuming and painful to edit CSS, but at least I had ample time to test new styles on QA before they were released to production.

One of my early hack day projects was to implement SASS. It was a long, hard fought battle, but at the end of the day, I had won.

Before I fell unconscious to the floor under my desk, I had chopped CSSthulhu into 16 neatly compiling .scss files:

It was a proud day for me. Possibly even the greatest triumph of my career thus far.

Once we turned on continuous integration, this meant that incomplete features, or features not ready for all users to see, were hidden behind feature flags. Now, you have to understand, I’d been spending the past few months creating standardized, semantic markup structures to be dynamically re-used throughout the application. These standardized markup structures don’t have any extraneous or unique ids or classes.

This gave rise to a few new problems in the UI:

1.) Once any CSS is compiled into the CSS build, it’s globally accessible throughout the application.

2.) Our C# feature flags won’t play with SASS. They won’t even give it the time of day.

3.) And now, as soon as I commit styles to github, it could be breaking the UI in production within just a few short minutes! After all, there is no unit test that can check for ugly. If the data is there, the build is going to say ‘Yay!’ and merrily pass my changes down the pipeline.

4.) Due to the unique nature of CSS, sometimes styles render and inherit a little differently on a local development environment than they do in production. Don’t shake your head, they JUST DO it’s one of the facts of life.

Therefore, when new feature UIs need to be built, they need new styles, that need to be insulated from the rest of the application to avoid CSS fights.

The first solution I came up with was to embed the styles inside a feature flag inside the Spark templates that generate the View. The problems with this were:

1.) Ewwwwww embedded styles?

2.) Why did I spend my blood sweat and tears building a beautiful immaculate SASS architecture if I’m just going to ignore it?

3.) I can’t use any of SASS’s sassy features inside the spark templates. Which means a lot more work, more lines of code, and general ugliness.

4.) Once features are launched, the embedded CSS has to be re-factored into the SASS build, which is time-consuming and risky to the application.

The ultimate solution was a compromise.

New features are placed inside a section tag with a unique class or id. This may not be perfectly semantic, but it allows me to develop new CSS in the SASS build with no risk to the rest of the application.

section.newFeature {

// a ton of new styles can live in here, with full access to the glorious mixins, partials, and targeted inheritances of SASS, while disallowing other parts of the application to inherit.

}

As new features are standardized into the application, these sections are removed, and the styles become standardized into the UI.

This might seem extremely obvious, but when you’re dealing with 50-60 dynamically generating page views, things just are never as simple as you would think.

Are there any other front-enders managing new feature development with continuous integration? We’d be very curious to hear about your processes to keep CSS safe and insulated.