Self-service ops

Exit if you you’re not in the mood for or allergic to rants.

I first wanted to title this post no-ops (acually fu-ops first), but then I found out they already had this discussion in 2012. So I went with self-service ops, which is less inflammatory and better captures my point. Toning down the title already used up all my positive good will for now, all the rest after this is complaints.

I think the software developer community tries to get out of their tech bubble and move closer to the end user for which they are actually building a solution.
Agile sets us up to cope with and respond to reality. DDD tries to find common ground and vocabulary to bridge the gap between the technical representation and business reality. There is still a growing attention to usability and user experience, moving beyond customer facing products to internal-only enterprise software.

On the ops side however,  I’m unfortunately encountering some system administrators in 2016 that only see abstract machines to keep running without engaging in a meaningful way with the business solutions deployed on them, which are the whole raison d’être of those systems in the first place. I’ll call these legacy ops or legacy sysadmins, since they represent a past we are working to leave behind.

When developers jokingly say “if it wasn’t for those damn users”, I’m thinking legacy sysadmins say “if it wasn’t for those damn applications”.

So what is the point of a sysadmin internal to a company, if any change to a company’s internal software solution has to come with a detailed instruction booklet? Are you really administering the system then or the janitor executing the cleaning instructions?

I just programmed the application, now I have to program the human being to install it for me?
The robots already took that job: meet Jenkins, Ocotpus Deploy, TFS Release Management and others. They’re faster, more reliable and don’t come with an attitude.


If a business solution breaks and stays broken on a testing environment, is it unreasonable to have expected any proactive action from the administrators of the system it runs on?
If not, what is their point?

If a new application gets developed, can we expect any interest into this application from the sysadmin that has to “administer it”?
I get no questions about its business functionality, dependencies, what to monitor, which resources to track for trends, no solutions offered to minimize deployment impact and ensure availability.
Just strictly the requirement to deliver a well documented (how to install it, not functional – don’t care), well tested (only want to install it once and never look at it again) application that won’t distract them or cost too much work.

This type of legacy ops is a useless obstacle in the way of getting things done. It frustrates the developer who gets obligations without help or rights, and the customer who already didn’t understand a simple change (tip of the iceberg problem) takes so long and now gets faced with rigid procedures to get value into his hands.

All this operates under the false dichotomy that either only legacy ops access a system and utopia arrives, or any trust given to non-traditional ops results in unstable systems.
On many occassions I have had my applications break, because ops performed changes to the system without collaborating with the developers that own business solutions on it (firewall changes are a favourite source for these types of issues).
So much for the false dichotomy.

Changing a company’s culture however, is not trivial – I would even say changing it deliberately is impossible. I think it rather organically evolves and changes without explicit direction (although I’m sure enough managers would like to think they control and steer a company culture).

So what to do?

I’m thinking self-service ops right now, build your needs right into your application and get it deployed once. Obviously there are limits to what you can administer from the inside from your application, but if you are creative you can drastically minimize legacy ops interaction to be happier and more productive.
At the minimum you need an endpoint to deploy new versions, after that you can incrementally add any needs you require.

Monitoring, alerts, tracking trends, support for A-B testing, can all be handled in software.
NoSQL (or a relation database with a custom strategy to store aggregates) can also minimize the need to require legacy ops for database schema changes.

I realize this is not a substitute for the real thing or a healthy culture with trust.
But I am saying it’s better than complaining and not getting things done.

Theme music for this blog post

Self-service ops