The Frenzy

So, what is a frenzy anyway?

Frenzy: A state or period of uncontrolled excitement or wild behavior.

That’s how things seem to be managed around here. Everything is an emergency. Everything needs to be done now. Everyone is trying to get everyone else’s attention all at once.

It’s the non-stop firefighting that sucks in everyone it encounters in the company. Without clear goals and good leadership the only substitute is to fix things when they as they break. It’s a directionless waste of energy that never really ends.

How can you fix it?

By having clear goals and good leadership.

By knowing what your goal is you can figure out a way of getting there. You can plan your strategy to avoid having fires to constantly fight. You can invest in training when you have the time instead of having to hack together everything because it’s being done at the last-minute.

Why is it like this?

My belief is that it’s because of the “flat org chart” that people brag about. The lack of structure leads to lack of accountability and empowerment. Without those you can’t have leadership and goals. All you have is a collection of people who are each trying to solve all the problems they see. That in and of itself is a good thing — but the lack of vision kills it. The vision needs to come both from the top, but also all along the chain of command. If there’s no command, how can you make decisions?

So what you have left is everyone scrambling around to plug the dike as things break.

The frenzy.

OODA Loop

I had a realization at work a few days back. My observation is that the decisions that are being made and the actions that are going on seem to be somewhat unrelated to what I perceive is going on.

This got me thinking about something I read online. I first encountered this was in relation to firearms and dealing with dangerous situations. It’s called the OODA loop.

This was developed by the US Air Force to deal with situations at all scales — of course this can be used in most situations as well.

From a high level the cycle is:

  1. Observe
  2. Orient
  3. Decide
  4. Act

This cycle continues without end. In the ideal case you’re running through this loop quicker than your adversary (regardless if they are a single person, or a rival army or corporation).

Each step is important.

We all go through life observing the outside world. It’s natural. The key is that to try to see the world for what it is without trying applying your own biases to it.

Orient takes into account what the objective is; where you’re trying to get to. It also takes into account the strategic situation you happen to be in — what your opponent is likely thinking.

Decide is where you figure out what you’re going to do. This isn’t necessarily the “right’ answer since many times that really can’t be divined. A decision must still be made none-the-less. You form a hypothesis to try and go with that.

Act is where you implement your decision.

Your action changes things which is why we go back to the top of the loop.

This goes on continually and iteratively we move forward to victory.

None of the steps can be ignored though. While everyone observes, it’s equally important that those observations go into the rest of the process. The decisions need to support the strategy. The actions must be consistent with the decisions. The orientation uses the observation as its input.

Too often I see the wheels come off though.

Decisions are made in a vacuum. Actions go contrary to the decision. Observations are ignored. In good times it’s easy to paper over the mistakes because of the rising tide effect. In bad times however this is a bigger problem.

Look around. Evaluate where you are and where you want to be. Come up with a course of action to try to get you there. Do it.

If you can’t because you don’t have the staff, well then there’s an observation. Do you need to do less? Focus your efforts more narrowly? Hire more people? Decide. Do.

A part of your business isn’t working as well as you would want it to? Pour more money in to shore it up and retrench? Pull out of it? Modify how you’re doing it? Decide. Do.

Not deciding or acting is itself a decision. It’s just a decision that’s the default. Know what you’re doing and be conscious of it.

Buck stoppage

We spend a lot of our waking hours at our workplace.

A week has 168 hours in. Let’s remove the time we spend sleeping: 168 – (7*8) = 112. Of the 112 remaining hours we are typically expected to put in 40 hours. If you’re salaried and exempt from overtime  it might be more, but the assumption should never be much more consistently.

To expect that we spend much more than that working or being expected to be thinking about it all the time is way too much.

Burn-out is real.

My iPhone is no longer sucking down work email. Neither is my iPad.

If the SHTF then I have web mail, but I’m consciously not going there unless that’s the case.

Why?

Because for the past three months the buck has stopped with me for just about everything. Responsibility is one thing. Responsibility and accountability without the ability to fix things is quite another. It’s accountability without control. Emergencies sprouting from people not taking my advice and consultation that leads the the buck landing on my head is demoralizing. Management by wishful thinking and relying on miracles is taking me down a dark road.

In this time my life has suffered greatly. I’ve been relying on crutches to make it through my day. I’ve been intolerable to be around.

I declared myself buck-free on Thursday.

I’ve felt better ever since.

I’m also weighing my options. Some options are already off the table.

Doing it your own way

It all comes back to the old saying “if you want something done right you have to do it yourself.”

That says a lot.

No one is as interested in how things turn out that you are. No one. Ok, maybe your mom or someone like that.

If you leave others to make decisions for you they might do some really idiotic things. It might not seem idiotic to them, but that’s because they’re not in your shoes.

This gets me to the office move.

If I had to do something like this I would ask one question: “What’s my budget.”

From there, I would do it right. Strike that, we would do it right.

We have to deal with the consequences locally.

No one except the folks in the office feel any of the pain.

Antipattern: Premature diagnosis

I preach against this, but got caught myself doing this. Taking a set of facts and developing a conclusion then using confirmation bias to make sure that’s the conclusion that wins.

Here’s the deal. We have a system that runs in the background on a server and pulls up customer records over time to send them reminder emails. Our system also encrypts a lot of the sensitive user info as well.

So things are going all honky-dory until one week when things go amiss.

  • We change up the client credentials (best practice to rotate these things occasionally)
  • We notice some errors from this system, specifically around encryption
  • Somehow the metrics associated with processing change

Ok, obviously we screwed something up around the encryption. So we futz around and make sure everything is up-to-date on that server.

  • Metrics go up again

At this point we’re spiking the ball in the end-zone. Problem solved, lets just backfill the stuff that errored out.

But the errors from the system don’t stop.

Huh?

Between fighting other fires (crazy week) we try to fix the obvious on-again-off-again encryption problem to no avail. No matter what we do the errors don’t stop.

Yesterday I finally had a chance to dig a bit deeper into the problem. Up until then I hadn’t had a chance to come up for air. While I was occupied another coworker that was in deeper on the encryption end had been chasing the problem when he had time.

What I found was interesting. The error that we were getting, wasn’t really an error; it was more of a warning when I looked at the code that was generating it. But since everything fit what we were thinking we just ran with it.

It’s a classic case of “ready, fire, aim.”

And it’s not that I didn’t know it before. This is a good reminder to validate your assumptions. Just because what you are seeing fits with your theory doesn’t mean you’re right. It’s always worth taking a step back to assess the situation a bit more even if you think you know what the problem is.

Now to figure out what the problem actually is.

<sigh/>

Live and learn.

…and then forget… and learn again.

Antipattern: Business owner by committee

This is some observations about another pattern that doesn’t work — when there is no clear business owner of a product.

When we started off we had a couple of strong decision makers. A committee of people would vie for his attention and the law was laid down. The buck stopped with the one owner of all things tech. It didn’t matter if you thought it was a good idea or a bad idea, the decision was made and we all moved on.

The fact that the buck stopped with someone is the real key.

As time passed and the organization became more diffuse this started to go away. Among other things a benevolent dictatorship isn’t very politically correct. Everyone needs a say into how the ship is going. With everyone having their hands on the controls the ship never turns. It’s just slowly cruises ahead with no direction. When the ship runs aground everyone has plausible denyability that it wasn’t their responsibility.

The committee is eager to claim responsibility for success and quick to denounce it when things go awry.

We tried a number of different schemes to manage this. We tried the scrum approach with different business owners getting their own teams. We tried the massive committee. Even anarchy.

Without clear top-down direction, things will rarely work right. When they do it’s a grass-roots victory from the trenches, not the top.

A clear direction gives the right priorities for a company. A person with whom the buck stops is empowered to make the hard decisions — because they are held accountable for the win or fail of their actions. Lacking that person the buck gets dropped, kicked to the corner and starts to collect dust.

I don’t think the buck can stop at a committee.

Antipattern: Testing, documentation, process, trust

Today I had the unenviable job of going through peoples’ desks to put things in the shredder bin. It’s not the job I want, but someone has to do it.

I saw a lot of old emails printed out, project plans, test plans and whatnot. But one stood out.

It was a solid inch think stack of paper in a gusseted file folder. I remember being on the other side of the testing for this bit of work a couple years ago. It took me roughly an hour or two to get my part of the job done. The fact that my hour or two caused someone to spend the time to produce roughly 200 pages of documentation shocked me.

I can see documenting things that have legal consequences. You need the signature to take to court if you need to go that far. I can see using notebooks to take, well, notes in meetings if that’s the way you’re most effective. Or a notebook to jot down ideas as they come to you. I will even allow for printing out documentation — because sometimes it’s easier than looking at it on a monitor. Or because you need to scribble on it check-list style.

But to consume that much paper?

Why?

The hour of work I did was to run a program with some new data. The program itself had been tested months before. It was known that the program already worked… the underlying issue seemed to be trust. Even though it worked every previous time, the exact same level of rigor was brought to bear on this new problem. Lack of trust led to gross inefficiencies.

Now on to the documentation part. In testing this project I can’t reasonably conceive how that amount of paper could possibly be used. Even if all of it was used in scratch work, there is no reason that the paper should have saved for the intervening two or three years. Again, the only thing I can think that would cause that is lack of trust in the system. To feel compelled to save the artifact of testing means that you don’t trust your managers (and the whole company in fact) to trust you that you did your job.

The two-fold CYA walks down a path of barely being able to get your job done.

Let alone excel.

A company that lacks trust in each other so thoroughly will build up processes to smooth the feathers of the non-trust. It doesn’t solve the problem (lack of trust), but simply layers process, the faceless entity that cares about no one, on top of everything. It doesn’t trust. It doesn’t not trust. It just is. It just slows you down.

Remember: You are all on the same team. If someone messes up, give them a second chance; you’d want the same if you screw up. Don’t walk so slowly that you can’t trip because then you’ll never run.

Fear of the upgrade

Many years ago we upgraded an Exchange server something something got messed up. Somehow we got in in our collective heads that failure isn’t an option.

That is a fail.

If you only ever go so fast that you never trip, you’re not going fast enough. You need to run fast enough that you sometimes fall down. Failure is how you learn.

By never failing you never learn.

People don’t typically get fired for messing up if it’s not intentional. You got to take some risks.

Get ready to fall down every once in a while. Pick yourself up and run some more.

Configuration files and tables. More antipatterns this week!

This is coming on the heals of yesterday’s post: the woeful configuration table that has too many rows.

Any time you need to configure something you need to make sure that at the end of the day things are configured as they should be. When you have a table with well over 70,000 rows in it, the chance that all of them are correct quickly approach zero.

Let’s examine how things can grow out of hand like that.

First off, lets make a config table for product line for instance. Maybe the color scheme for each product.

Product Line Color
Vacuum Cleaners Blue
Computers Red
Radios Black
Phones Yellow
Camping Equipment Fuchsia
Tea Puce

This is all good and a normal, non-OCD, human can go in and manage this table just fine. It’s something that works. (Also imagine another 100 product lines in there. I’m lazy enough to not want to make something that big) Additionally let’s assume we have two or three more columns of configuration on each of these rows.

Now the business comes to you and says “this works fine except for customers coming from Apple really should be aquamarine.”

Now what?

Two solutions come to mind:

  • Make a special case in the code for Apple’s source (42 in this case)
  • Add a column to the table for source and multiply the number of rows (product lines) by the number of sources.

The first one sucks, the second one is is horrific. Lets go over each one in turn.

Adding a special case with an ID in code is bad. It makes something which is a database abstraction and moves a hidden dependency into the code itself. Now “42″ is magic. (Even more magical than before) Just by looking at the configuration table you’d have no idea that Apple’s traffic is yellow. Moreover the code is now tied to the database as well; what if the test and production databases are different for some reason? Even worse the precedent has been set that this is the way to do this type of thing. The next novice coder that comes along will probably do nothing except copy it and make “105″ just as magical later on. You can see this going down an slippery slope.

The second choice might seem like the better option at first glance but it’s evilness runs even deeper. We have a hundred product lines and a thousand sources. Now we have 100,000 rows in the configuration table. OMG! This might work in the short term when all we do is script out a change… but in the long term this will fall on its face even harder than the first option since this creates so many rows of data that you can’t maintain that many with any reliability.

Given just these two options, the first option is probably the better one. But it’s still pretty sucky.

Really, what you want is some variety of decision system that won’t automate the process, but allow for the flexibility to come up with the right answers. The key is to make sure that you have the minimum amount of data to encode the required information. What you want is to make sure that you don’t store any more redundant data than is absolutely necessary.

Not having a system like that, I’d make my own language to support it in some way.

Based on this if I didn’t have the luxury of a decision system or to make something myself I’d hold my nose and do number one. Sucks, eh?

CYA – How to make it suck even more

Last night we were working to fix a bug that was about to go into production. This isn’t about the problem with how our system is configured; that’s for another day. This is about the thought of dealing with the change.

We had to make a change to a lot of rows in a config table. A little more that 1300 to be more precise. Thankfully I was able to come up with a scheme that’s automated so I wasn’t entering these rows by hand. Even so, whenever you’re making a change to a production system you have to wonder if you’re doing it right. After all that’s a lot of data to be mucking with. The question I have is “how do I know I got all the data correct?”

The suggestion was to email the list of 1300 IDs to make sure they’re correct.

!

Homie don’t play that.

I’m not going to put someone in the position where they can’t succeed. That’s just not fair. The only thing that does is play some illegitimate CYA game. When things don’t work you have something to point to and say “it’s all right here.” As if someone can check everything and make sure it’s all correct. If anything the act of checking it all would lead to even more errors and confusion. Not cool at all. To expect someone to go through the list of all of those numbers to verify that they are all correct is expecting the impossible.

On the other end of the chain Ennie sometimes gets those lists thrown at her. When things don’t match up in their reporting system she hears: “Why didn’t you check that list. That’s why we sent it over to you.” Of course it’s actually more fun than that, the only way to check the list is from the list itself. Umm… Tautology, here we come!

No one should be expected to make that work. You can’t.

If you’re given something to work on, you should have at least the remote expectation that you can get it done. If there’s no way, it’s probably nothing more than someone playing this stupid game. Make sure you call them on it.