ALL BLOG POSTS AND COMMENTS COPYRIGHT (C) 2003-2016 VOX DAY. ALL RIGHTS RESERVED. REPRODUCTION WITHOUT WRITTEN PERMISSION IS EXPRESSLY PROHIBITED.

Tuesday, May 26, 2015

The principles of Agile

I've never been much impressed with any of the "best practices" concepts, from Six Sigma to Agile. They strike me primarily as a way for midwitted bureaucrats and technical workers of modest talent to cover their asses and claim failure can't be their fault because they are Doing Everything The Right Way:
The Twelve Principles of Agile Software
  1. Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.
  2. Welcome changing requirements, even late in development. Agile processes harness change for the customer's competitive advantage.
  3. Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale.
  4. Business people and developers must work together daily throughout the project.
  5.  Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done.
  6.  The most efficient and effective method of conveying information to and within a development team is face-to-face conversation.
  7.  Working software is the primary measure of progress.
  8.  Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely.
  9. Continuous attention to technical excellence and good design enhances agility.
  10. Simplicity--the art of maximizing the amount of work not done--is essential.
  11. The best architectures, requirements, and designs emerge from self-organizing teams.
  12. At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly.
My thoughts on each point:
  1.  Sounds good in principle. In reality, the customer seldom knows what he really wants. The good designer has to anticipate his user and provide him what he doesn't know he wants yet. "Early and continuous delivery" sounds like more ass-covering stuff. "It's not our fault, he approved the deliverable" is what this sounds like to me.
  2. Bullshit. That's fine in the early stages. In the middle to late stages, this is what is known as "mission creep".
  3. Good for testing, but against, strikes me as more CYA, especially if it is going out to the customer.
  4. No. Hell no. While the biggest failure of which I've been a part was the fault of the chip engineer failing to listen to the marketing guy, I did talk to him on a bi-weekly basis. Talking to him daily wouldn't have helped. The key is that the technical people must LISTEN to the business people, not see their faces on a regular basis.
  5. Yes.
  6. No. I run into this problem frequently. Most companies would rather have an inferior employee they can talk to face-to-face than a better one who is external. This makes no sense, especially if they then leave the position vacant because they can't find anyone local who is good enough.
  7. Yes.
  8. Dubious. Sounds like snake oil to me.
  9. True, but obvious to the point of being tautological. "Program good code that works properly!" Okay....
  10. True. But I prefer "don't reinvent the wheel".
  11. No. A certain amount of flexibility is desirable, but there must be someone who is accountable for the vision and someone to make the hard decisions. Which is to say a designer and a producer. The best designs most certainly do NOT emerge from self-appointed committees.
  12. Yes, this is reasonable.
Overall, the entire concept stinks of being little more than client-marketing to me. "Hey, we're AGILE-CERTIFIED, obviously we are much better than those guys over there who don't have CREDENTIALS. All they have is a proven track record of delivering successful products, and we all know how little that means.

Labels:

190 Comments:

Anonymous RS May 26, 2015 7:09 AM  

The value in these improvement frameworks are usually the ability to set clear expectations throughout the org. Kind of like Robert's Rules - they aren't perfect or even really all that great and won't make terrible people behave in a civil manner, but allow people who are trying a framework to successfully interact.

Changing reqs doesn't mean scope creep, it means that business requirements can literally hangs overnight and shouldn't take a 3 year planning process to integrate into software. I think most of your criticisms are equally applicable to other development frameworks, and doubly applicable to an org not using one.

With that said, committed and high achieving people can certainly function fine without them.

Anonymous Smokey May 26, 2015 7:16 AM  

Principle #2 is basically the main reason why 80% of IT projects are stuck in development hell. Change, change, change, and no concrete focus. Almost at the finish line, then it gets placed a mile ahead again.

Blogger Ron Winkleheimer May 26, 2015 7:19 AM  

#2, mission creep, is problematic because once a customer figures out that there is no push back on requiring new features they will start demanding things without offering any more money or time. That is why a well done requirements document is so important. In addition, the later in the stage that you introduce a requirement, the harder it is to integrate it into the design. You end up with a lot of cobbled together crap.

That said, some process is going to be needed to add new requirements because:
1) No requirements document is going to be perfect.
2) Requirements do change.

Vox is absolutely correct that the customer usually doesn't know what they want or need and that it is necessary to anticipate their needs. But I don't think there is anyone who has worked in software more than 3 months that doesn't know that.

Anonymous Smokey May 26, 2015 7:19 AM  

Changing reqs doesn't mean scope creep, it means that business requirements can literally hangs overnight and shouldn't take a 3 year planning process to integrate into software.

If we mean minor changes, then this might be achievable. Otherwise, late in development, overhauling major concepts almost always means more headache for the team, and unwanted mission creep. There rarely, if ever, is a painless way to get through this.

Blogger Ron Winkleheimer May 26, 2015 7:22 AM  

A good requirement document is also necessary so that the customer can't request a new feature by claiming that its absence is a bug.

Blogger JACIII May 26, 2015 7:23 AM  

+1 on the CYA, but add to that the continuing effort to refine management of anything by anyone into a process that requires absolutely no familiarity with the product or work being done.

It reminds me of a series of "How to" music books from the seventies: "How to be a fake (drummer/violinist/trumpet player,etc"

Six Sigma, Lean, DFT, Agile, etc = "How to be a fake manager"

I see engineers lapping this shit up and think, "Kiddo, we could go pick out a random 9th grader to do this shit better, faster, and cheaper than you and he would be far less annoying"

Blogger Crowhill May 26, 2015 7:24 AM  

My limited experience with development shops that adopt the Agile concept is that they're more interested in the fact that they're Agile than they are in doing the project.

Anonymous Stg58 / Animal Mother May 26, 2015 7:26 AM  

Systems like these work well in oil and gas technology to a certain extent. There are limitations related to laws of physics that obviously can't be circumvented, so sales people must be the link between customers and the factory to ensure what the customer wants can be done, and any mid project changes can be tracked, evaluated and accommodated.

If you don't do this, the equipment shows up and never works, then you get a bad reputation in that conpany, which will kill you there. The guys that operate our equipment are rednecks and good old boys, not tech software users and programmers.

Anonymous zen0 May 26, 2015 7:26 AM  

@ Smokey

> Principle #2 is basically the main reason why 80% of IT projects are stuck in development

Didn't Henry Ford and Bill Gates get rich bcz they did't fall into that trap?

Blogger Nate May 26, 2015 7:32 AM  

This looks like a list of Feelgoods designed specifically to give suits a buzz.

Anonymous Stg58 / Animal Mother May 26, 2015 7:36 AM  

Yes, these processes look good to accountants and MBA zombies, in my opinion the bane of American business. If the MBA went away we'd all be better off.

Blogger Nate May 26, 2015 7:39 AM  

managers are a liability... not an asset. Any business that grasps that has a built-in competitive advantage.

Blogger BCM May 26, 2015 7:40 AM  

#7 should be "working software which matches correct requirements".

What do you think of the core idea of agile, delivering smaller pieces frequently as opposed to the larger scope delivery of a longer duration? (I don't think the manifesto communicates the point about smaller deliveries well. )

Blogger grendel May 26, 2015 7:44 AM  

How the hell did anyone get anything done before some desk shrub invented "Teamwork" and began stuffing it into every sentence? Maybe soon we can leverage our synergies. ISIS will be in trouble then!

Blogger Ron Winkleheimer May 26, 2015 7:46 AM  

I can't tell you how many classes I have taken on the latest management fad. I mean I literally can't because I have been to so many. ISO 9001, something that was big in the 80s when I was in the Army, something about rapid software development where the customer (someone who would actually be using the software) would sit with the developer and look over prototypes. There are to many to remember.

There is a cycle in software development. It swings between strict requirements and a very structured timeline with deliverables on certain dates to a much looser "we are all in this together and we are going to produce world class software by working together to overcome obstacles rah, rah, rah" project structure.

The reason they keep switching between the two methods is because both have advantages, but they have disadvantages also. Anyone who has ever read a book on managing a software project would know that. But apparently few people do so.

Anonymous NorthernHamlet May 26, 2015 7:53 AM  

On 6, everyone knows the saying "get it in writing." There's no easier way for someone to play the memory hole game than to take anything they say aloud seriously.

Blogger fisher man May 26, 2015 7:59 AM  

This is agile development after middle management took it over in order to maintain their roles.

That list is nothing like it was when it started. In fact it was a backlash against those sorts of rules.

It was about productivity first and formost, by dumping the incessantly interfering management. It gave the developer autonomy again, and responsibility for getting stuff done on time.

Being purely a meritocracy, it had to be destroyed because only good developers succeeded.

Blogger IrishFarmer May 26, 2015 8:00 AM  

Agile would be all right, but 50% of the sprint (on short sprints) is spent having meetings to prepare for meetings or creating artifacts to prepare for the meetings that prepare you for the other meetings. Seems highly wasteful, although the stakeholders and managers absolutely love it.

Anonymous MrGreenMan May 26, 2015 8:01 AM  

The problem with #2 is if billing is not structured correctly. Let's suppose you have a three-installment project, scoped to 16 weeks (So a good 4 months; opinions are going to change). Here's two scenarios for the little software shop that could:

1. You get a single document at the start of the four months that specifies what it must do at the end. You deliver it for testing and review in week 13. You spend weeks 14-16 fixing bugs and nonconformity.

2. You get a rough idea and decide you are going to structure it in one week sprints, meeting with the customer every Friday and willing to accept new requirements. At the end of week two, the customer has some internal turnover and somebody else who wants to put his finger on the project does so. You as the developer are told that, despite what the last guy communicated - orally, in the meeting, the best way to understand, mind you! - since it's not in the contract, you have to do it this new way to get paid for the project. You now start again with week three as week one as al of the first two weeks' work product goes in the trash.

In case #2:

a) Is there adjustment to move it from 16 weeks to 18 weeks?
b) Is there an increase in the amount that the project is going to pay?

Because, as an outside developer, I have never once encountered the enlightened philosopher customer who understood that, because he changed his mind, you should be compensated. Rather, you will have an angry customer who thinks not that he changed his mind, but that you misunderstood, and it's your fault and you should suffer for it being wrong and late.

Agile can compound this sort of downward spiral. It's great when you consider it to be a giant software department within a giant corporation, and the IT director is going to go to the mat for you to get things extended, and the PM is going to shout down the business users who say they want to maintain quality and timeline while adding features -- but this is not reality.

Blogger VD May 26, 2015 8:02 AM  

What do you think of the core idea of agile, delivering smaller pieces frequently as opposed to the larger scope delivery of a longer duration?

You mean milestones? Which we were using long before Agile ever existed? That's literally how I've always done software development. What I think is you've just convinced me that Agile is nothing but marketing.

Blogger fisher man May 26, 2015 8:11 AM  

This comment has been removed by the author.

Blogger Éibhear Ó hAnluain May 26, 2015 8:12 AM  

Principles #10 and #11 conflict with each other, particularly with new agile teams and *especially* when the senior developers fancy themselves are architects and are more self-confident than self-aware.

I've seen robust architectures and efficient delivery processes turned into unmitigated messes by such "self-organised" teams.

The excuse: but, but, but, but they're not appropriate for agile projects!

My response: Well, what have you got as an alternative?

It turned out that declaring we have agile in operation was more important than maintaining the stability of the systems.

Anonymous MrGreenMan May 26, 2015 8:16 AM  

Agile is nothing but marketing

It sure seems that way. There are some good ideas that sometime travel with agile, but they are not in that list:

a. Reduce programmer meeting overhead. I remember hearing a lot of this when this started, with your morning meeting, your end of day readout, all in all, about 1/8th of the programmer day in meetings. Of course, as more of the bodies start "doing agile", well, the meetings just happen. I remember being somewhere where the developers "did agile", and we had one 15 minute morning and one 15 minute afternoon meeting. However, once we got a VP of Operations, a CTO, a VP of Product, etc showing up for these meetings, suddenly it was 1 hour at the start, 1 hour at the end, all day Wednesday, Friday morning, and Monday afternoon - with the CEO showing up for the Monday and Friday ones.

b. Produce regular work product. This actually is a good practice for managing and firing the bad programmers. If they can't do something in a day, they can't do something in two days. Sometimes it's just a written plan, but something that looks good. Of course, stick them in face-to-face meetings all day, they are going to produce nothing.

c. The point of the quick meetings at the start and end of day was supposed to be that the PM had a checklist and could see - was what was expected yesterday done; what is expected today - then the PM can go back to Microsoft Project, check or extend the little tasks, watch that roll up to the big task, then email a Gantt chart with updated new timelines to the people who authorize payments for accountability.

All of these things exist outside of the "Agile" label if you use what should be simple ideas of:

1. Programmers should be programming, not meeting all the time.
2. Structure the project for accountability at all levels, otherwise you're going to be surprised when weeks pass with nothing getting done.
3. If you want to prevent the main timeline from slipping, you have to ruthlessly decompose into small tasks. Programmers can estimate if something is an hour, a day, or maybe two days. Beyond this, programmers are incapable of estimation. The daily meetings are just a score whether these estimates are right. It does mean that things have to be much more understood than the regular "Here's what I got! Am I close? What do you want me to do next?" meetings would suggest.

Blogger sykes.1 May 26, 2015 8:17 AM  

A number of years ago, my department at Ohio State was caught up in the vision/mission/competency thingy. I spent one awful afternoon in a department meeting where my colleagues thrashed out the needed statements.

When it came to competency, various colleagues proposed items that we were obviously excellent at and that should be advertized. The resulting list, duly approved by majority vote, was a pack of egregious lies. But most people present were happy to wallow in them like pigs in mud.

Blogger fisher man May 26, 2015 8:21 AM  

I hate typing on a phone..

Anyway the original points of agile development (before management destroyed it) was unit testing, and much shorter times between milestones.

So each iteration focused on less stuff, and the unit testing improved the quality.

You need competent developers though, which are rare. Some types of software are incompatible with unit testing, and thus do not work well with an agile method.

Anonymous Leonidas May 26, 2015 8:21 AM  

Never been much impressed with Agile, either. But rather than marketing, which it probably also is, it always struck me as a way for mediocre programmers to adopt a process that would allow them to pretend that they could actually keep up with the really good programmers.

My principles of programming (or any kind of development):

1) Hire really good people who know what the hell they're doing.
2) Get the hell out of their way.

Anonymous MrGreenMan May 26, 2015 8:26 AM  

9. Continuous attention to technical excellence and good design enhances agility.

You know, I've often been the architect, and here's the problem: Have you ever built a house by first plopping down the kitchen, putting a couple doors on it, and then asking - and this door - does it just go outside - or should it go to another chamber?

I have seen a house built like this. It was a work of art that was unmaintainable by any but the craftsmen who built it. Further, it lacked certain things - the HVAC never worked right, and the plumbing was very challenging, as was the electrical. Further, some clearances were not exactly right because they had to be worked into the structure once built.

Agile sometimes goes so far as the idea of pushing up from the bottom can always work. I think it's an overindulgence in the theory of evolution - just keep making little things, then combine them, and keep pressure from the bottom up. The problem is, without some serious guiding principles, top down, you are going to make decisions that, in hindsight, seem atrocious because they aren't made with the entire picture in mind.

For example: You might start using a NoSQL database just as a fast key-value store. However, you keep pushing on that piece, you wind up storing things that start to look relational, and suddenly you're writing business logic controls to do what a RDBMS does for free out of the box, and you've lost the idea of your fast key-value store altogether, because you let things grow organically and satisfied the objective for the next meeting, not the big picture.

Anonymous JamesV May 26, 2015 8:28 AM  

Agile is like communism. The reason it doesn't work is because no one has done it right...yet.

I've suffered under agile for the past 5 years and I'm ready this trend to die a horrible death.

Anonymous MrGreenMan May 26, 2015 8:31 AM  

@Leonidas

But rather than marketing, which it probably also is, it always struck me as a way for mediocre programmers to adopt a process that would allow them to pretend that they could actually keep up with the really good programmers.

There's truth in this; several of the places that I saw crash and burn trying to do Agile were also the places wherein I was introduced to the term "programming resource" instead of programmer. There is some presumption that by some sort of process, they can make all programmers like assembly line workers, which really means hamstringing the ones who are truly brilliant.

Anonymous mjf May 26, 2015 8:34 AM  

Been on several agile projects. They mosty worked at the small shops, but in the corporates were a disaster.

Whatever happened to XP and Kent Beck with fear is the cause of all project failure?

Anyway software development is 'kin easy. JUST USE GOOD PEOPLE. Of course recruiting good people isn't so easy...

Blogger dc.sunsets May 26, 2015 8:42 AM  

Taylorism at it's most bureaucratic.

Frederick Taylor's Scientific Management is an artifact of the Progressive Era and arguably best suited to static industries enjoying such complete regulatory capture that upstart competitors are of no concern at all. Applying it to any competitive industry may be a signal that entrepreneurial activity is being crowded out by a public-private cartel. Or it may simply signal which firms are the dinosaurs.

Any kind of Best Practices regime is just another bureaucratic substitute for profit/loss signals determining what "practice" is best, and one more illustration that small minds can't grasp the message of Leonard Reed's "I, Pencil."

Blogger fisher man May 26, 2015 8:45 AM  

Good developers are the most important. They are also impossible to find. College no longer produces them, so you get lucky to come across one.

I usually interview more than a dozen for any position, then eventually decide to do it myself (and move out the deadlines)

Sometimes it is so frustrating to interview these idiots that I cannot help but tell them they wasted 6 years getting a useless phd, or masters. I have actually had developers cry in interviews after explaining why they are useless.

Anonymous RS May 26, 2015 8:46 AM  

This reminds me of the soccer teams I played on when I was younger. The talent on my club team ranged from excellent support players (me) to superstars, and everyone was pretty smart and approximately the same level of experience, so our coach was really more of coordinator than anything else. We saw a lot of success. My high school team (which featured a few of the same players) had a way larger talent, intelligence, and experience range.

Of course, this lead to the superstars on the HS team (and people who thought they were superstars) to get frustrated, complain, and dick around when our coach was installing a new base formation, moving away from boom-ball (which the club team was skilled enough to play effectively, HS team was not), and provide enough training in practice to allow kids who sucked to come along. I plead with the superstars (and faux superstars), saying "yeah, not everything coach wants us to do is smart or even right, but I promise we will be better as a team and win more games if we all pull in the same direction" - of course those pleas fell on deaf ears. Yeah, one or two of the superstars would have been right at home at an ODP or state/region level elite club team, but they were absolutely terrible at working as a team.

As with any framework, its as good or as bad as you make it. Yes, if you let your customer push you around and unseat you as the developmental expert, you're going to have a bad time. Yes, if you put Sally Eeoc through "black belt" training without ever stepping foot on a production floor, then let her loose to pursue her own projects, it's going to get ugly. But if you have a well meaning team with differing opinions on how to get something done and need to investigate a problem and propose effective solutions, Six Sigma is a great way to do so. Does an industrial engineer need Six Sigma? No, it's there to make sure the rest of the team can follow along and step in the correct direction.

Basically, this is Vox's "I'm way smarter than everyone so it's hard to relate" thing in action. These frameworks seem really dumb to smart people until they've been forced to do project work or process improvement with not-as-smart people.

Blogger Nate May 26, 2015 8:49 AM  

"These frameworks seem really dumb to smart people until they've been forced to do project work or process improvement with not-as-smart people."

Look... the world managed to move along just fine without these buzzword ridden feelgood processes. If you need this sort of hand holding... then you should consider revising your hiring practices.

Employing a bunch of window lickers is no way to run a business.

Anonymous Mike M. May 26, 2015 8:52 AM  

My experience, over 35 years in flight test, has been that every one of these management fads du jour has been a way for second-rate managers to pretend they are competent.

Anonymous RS May 26, 2015 8:54 AM  

Nate,

I agree and thought about that right after I posted.

Sounds like Vox and some of the other commenters have never been in a situation where they HAVE to work with window lickers.

You'd be surprised at how few people don't lick windows, though. Remember, MPAI.

Anonymous AlteredFate (#92) May 26, 2015 8:56 AM  

All of these "quality" and "productivity" systems are nothing more than a money grab by a cabal of consultants and certification orgs who have created a pay-to-play system where businesses hand over tons of money in exchange for a stamp of approval which means nothing more than their business can navigate and internalize bureaucracy. The corporation is a person, and this is the equivalent of their modern college degree.

Blogger RC May 26, 2015 9:07 AM  

Kelly Johnson's Skunkworks provides evidence that small, highly capable teams are the most productive. Very intelligent people in small focused teams just make things work.

I had the pleasure of working with a brand-new wireless startup funded by some major players. At the time the group consisted of fifty people, the majority were RF engineers, massively funded, and they just needed tools that worked. It was a great ride helping these guys build new tools and systems, need it now, hunker down, nights, weekends, get 'er done. Small, focused, motivated teams of exceptionally capable people will always find a way to get the job done.

My current company adopted Agile because the product was a mish-mash when I purchased it and we needed to demonstrate that a stodgy old product would be rapidly improved, so the sprints have been effective in that regard. But, once again, small teams of exceptional men is the most important component.

Bottom line: If you're capable of accurately evaluating talent, can attract those people to your team, can motivate and retain them, you'll win. The rest is superfluous.

Blogger Nate May 26, 2015 9:13 AM  

"Sounds like Vox and some of the other commenters have never been in a situation where they HAVE to work with window lickers."

we've all had to deal with window lickers. that's the world. But the idea that a company would want a certification that basically says "Look we have a process that minimizes the impact of our window licking staff" is really sort of dumb in a marketing sense.

particularly since the process itself indicates a basic ignorance of the software world.

Blogger Nate May 26, 2015 9:17 AM  

"Bottom line: If you're capable of accurately evaluating talent, can attract those people to your team, can motivate and retain them, you'll win. The rest is superfluous."

correct. if you have that... you don't need agile or anything else. you need clear goals. You need static project vision.

And by that I mean an end plan.

Imagine a contractor building a house trying to employ this sort of idiocy. EMBRACE! CHANGE! even in the late stages!!!

Really?

So the house is supposed to be done two weeks from now... and its all framed up and you're putting in the last bits of flooring and finishes... but hey... ya know... shouldn't the bathroom be over there? And the kitchen be over here?

Sure! embrace the change! what could go wrong?

Blogger Josh May 26, 2015 9:22 AM  

This strikes me as marketing specifically targeted at buyers who have no business actually making decisions on which firm to hire for a project.

Blogger Nate May 26, 2015 9:22 AM  

"This strikes me as marketing specifically targeted at buyers who have no business actually making decisions on which firm to hire for a project."

exactly.

feelgoods for suits.

Blogger VD May 26, 2015 9:32 AM  

Sounds like Vox and some of the other commenters have never been in a situation where they HAVE to work with window lickers.

I do my level best to always avoid it. I simply won't work with people who aren't at least highly competent.

The thing is, the vital decisions are always the ones that no process can solve. The vital key of our Intel success was a single decision. Redbook audio or the new Direct X audio. Literally everyone else trusted Microsoft and was screwed when Microsoft moved back the delivery date. Big Chilly and I were the only ones who looked hard at the option and asked ourselves: when has Microsoft ever delivered development tools on time?

Process is never going to solve those issues.

Anonymous Rolf May 26, 2015 9:36 AM  

WRT #4 - the biz people need to listen to the technical, too. If the biz guys are demanding magic, call them out on it, and make them understand when you can't get from here to there not because you are lazy, but because it defies the laws of physics.

Blogger swiftfoxmark2 May 26, 2015 9:39 AM  

This looks like a list of Feelgoods designed specifically to give suits a buzz.

You have no idea.

As a software developer who has to work with this, I can verify VD's observations. The main problem with Agile is the Story Point system where tasks are given weight based on an arbitrary point system. Unfortunately, this point system is about complexity, not necessarily the time it takes to complete the task.

But then when you assign the tasks to developers, you have to use the time it takes to complete the task as a factor.

Makes sense right? You assign tasks points based on complexity not duration, but then you assign the tasks based on time and capacity.

Talk about utter nonsense.

The funny thing is, Agile is supposed to be about flexibility and is not a set of rules follow during development. But at the same time most managers enforce them like a hard set of rules.

Blogger VD May 26, 2015 9:43 AM  

If the biz guys are demanding magic, call them out on it, and make them understand when you can't get from here to there not because you are lazy, but because it defies the laws of physics.

Sure, as long as the tech guys aren't being morons and running their "knee-jerk no" routine, which only makes them look like incompetent idiots when they tell the business guys that it is impossible to do what someone else is already doing.

If there is one thing that irritates me about tech guys, it's when they say "well, how do you even know X is possible?"

Because I always do my fucking homework. Why the fuck do you think I'm TELLING you it is possible rather than ASKING you if it is? I KNOW you're more technical than I am. You don't need to defend your MOST TECHNICAL BOY IN THE ROOM title every single time I talk to you.

For all that they pride themselves on their logic, I've found that technical folks aren't much better about applying it to their own behavior than the average teenage girl.

Blogger szook May 26, 2015 9:43 AM  

Now...now...now, Vox! Next you'll be expressing your heretical skepticism about the Cloud.

Blogger Stilicho #0066 May 26, 2015 9:43 AM  

so sales people must be the link between customers and the factory to ensure what the customer wants can be done


What would ya say...ya do here?


Well-well look. I already told you: I deal with the god damn customers so the engineers don’t have to. I have people skills; I am good at dealing with people. Can’t you understand that? What the hell is wrong with you people?

Anonymous BigGaySteve May 26, 2015 9:49 AM  

buzzword ridden feelgood processes. If you need this sort of hand holding... then you should consider revising your hiring practices.

How do you explain it to the EEOC? Holder & Lynch both agree you should be hiring felons with credentials from Haiti University.

2.Welcome changing requirements, even late in development- no one can honestly say that
7. Working software is the primary measure of progress- sounds merit based to me
8. ... The sponsors, developers, and users should be able to maintain a constant pace indefinitely.- Is infinite memory available now. I remember one sales pitch was for the ability to detect mistakes that had been corrected for a longer period of time. " So what you are saying is we can hide malpractice and it wont be provable 2 weeks later unless we buy an upgrade?"

Anonymous Athor Pel May 26, 2015 9:51 AM  

"RS May 26, 2015 8:46 AM
...
this is Vox's "I'm way smarter than everyone so it's hard to relate" thing in action. These frameworks seem really dumb to smart people until they've been forced to do project work or process improvement with not-as-smart people."




This subject is a perfect example of midwit versus very high IQ and the disconnect between them. The mental models we use for how other people think can be terribly inaccurate. Being as how we base those models on our own minds it stands to reason it would be this way.

What I see going on in software development is the presence or absence of the ability to predict consequences far enough into the future. The average programmer/designer works according to a plan, modifying things as circumstance requires but never seeing around the corner, never seeing beyond two or three steps in a cause-effect chain. The smart ones foresee danger and plan accordingly, they see around the corner and sometimes that corner after that.

Trying to convince your boss that you need to spend x+2 hours and x+3 dollars now instead of x+10 and x+15 later becomes impossible when they are incapable of imagining the future you describe to them or the "good enough right now" result is a core operating principle of the business.

My personal theory, too many people believe in their gut that computers are magic boxes and those same people believe their birthright is to have one button applications that do what they want rather than what is asked.

Anonymous Mike T May 26, 2015 9:52 AM  

In my experience, agile seems radical to people who are accustomed to this sort of process:

1. Get large bucket of institutional funds (be it corporate or government).
2. Find company willing to do work.
3. Load funds into catapult.
4. Fire!
5. Come back a year later and see to what ends the funds have been used.

It feels revolutionary because agile (lower case "agile," much like "lower case libertarian") is really a polite way of telling the "product owner" to get off their ass, take responsibility for deliverables and actually communicate with the people they've hired more directly than smoke signals and augurs.

Blogger Stilicho #0066 May 26, 2015 9:53 AM  

Now...now...now, Vox! Next you'll be expressing your heretical skepticism about the Cloud.

One of the most valuable services I ever rendered a client was convincing them to not use cloud-based storage/retrieval for their business data. The security concerns far outweighed the cost of maintaining their own servers and back up storage. Sometimes, unconsumated deals are the best deals.

Anonymous Will Best May 26, 2015 9:54 AM  

#6 seems more like an anti-Indian code shop principle. Either that or an argument to outsource all of development to India.

Blogger Ron Winkleheimer May 26, 2015 9:54 AM  

The ultimate goal of most of these management fads is to make it so you don't need to hire good programmers. Good programmers are expensive. Its better to bring in a bunch of not really competent h1b visa holders to reduce costs. The process will compensate for any deficiencies in aptitude and experience. After all, its not like programming is all that hard. If programmers were really smart they would be in management.

True story. Once read an article where a woman, managing a team of programmers, noticed that a lot of compilation errors involved typos. So she wanted to improve her team's productivity by having the programmers take typing classes.

Blogger Cail Corishev May 26, 2015 9:55 AM  

7. Working software is the primary measure of progress.

This one is so clear and fundamental that I'm wondering how it slipped in there. Most of the rest read like buzzword bingo, a mix of obvious and meaningless.

Because, as an outside developer, I have never once encountered the enlightened philosopher customer who understood that, because he changed his mind, you should be compensated.

Right. While a strict requirements document sounds great, as an individual developer, I don't know if I've ever had a client who could have provided one at the outset of a project. (A client who had done his homework, as Vox says, and knows what can be done and what he wants, sounds like a dream.) Usually their requirements amount to something like, "I want a site that can do X, kinda like what ABC is doing. I'll know it when I see it." They may only be able to describe 20% of what they really want. That fits my style well enough, since I'm a jump-in-and-figure-it-out-as-you-go type of guy by nature, but it's heck getting paid enough for it. Since it's just not possible to start with a requirements document that resembles the finished project, you must either: A) build in an extra rate that applies to everything that gets added, which most clients won't like when the "extra" ends up costing 5 times the original estimate, or B) quote the job at 500% and hope that's enough.

Anyway, enough rant: that kind of adjust-as-you-go flexibility is hard enough when you're a one-man shop; it seems like it'd be a real mess with a larger team. And once you're having daily meetings with each other and with outsiders, it's hard to see how anything ever gets done. Probably most of it gets done by whichever guys manage to avoid the meetings.

Programmers can estimate if something is an hour, a day, or maybe two days. Beyond this, programmers are incapable of estimation.

So I'm not the only one? That's a relief.

Anonymous Drew_Deuce's May 26, 2015 9:57 AM  

My one (& only) experience in hiring a tech team for software development involved agile methodologies. The process did work well and was able to withstand our various changes as the project progressed. Instead of just marketing, isn't this a way to standardize process(es) across the industry? The alternative would seem to be a crap shoot based on the approach & abilities of each team (from the perspective of one not coding, but rather looking for a person/team).

We've piloted our program and are now setting to launch our startup. Grocery Heroes

We do have room for anyone interested in partnering in the next phase.

Blogger Stilicho #0066 May 26, 2015 9:58 AM  

It feels revolutionary because agile (lower case "agile," much like "lower case libertarian") is really a polite way of telling the "product owner" to get off their ass, take responsibility for deliverables and actually communicate with the people they've hired more directly than smoke signals and augurs.

It's a process checklist of sorts for people who cannot be trusted to do their jobs without one. Even then, it will only work to the level of the people implementing it. Beyond that, I don't see what it accomplishes other than what Vox pointed out re: credentialism.

Anonymous DeepThought May 26, 2015 9:58 AM  

@RS - "Changing reqs doesn't mean scope creep, it means that business requirements can literally hangs overnight"

As a manager at a fortune 50 company's IT division, I have to question RS's comment as nonsensical and not based on my experiences as a consultant and manager in IT development and support over the past 20 years. Major scope changes at the end of the development life-cycle will = failure or increased cost.

Blogger HalibetLector May 26, 2015 10:01 AM  

How do you guys do software development then? The chaos approach? Aka: hire talented people and let them do what they do however it is they do it?

As a humorous note, Zed Shaw gave a great talk in 2012 about how these software development fads that pop up every 10 years are sold using a variation on communist propaganda.

The talk, for anyone who's curious: https://www.youtube.com/watch?v=c5Xh2Go-jkM

Blogger maniacprovost May 26, 2015 10:02 AM  

Vox, ever read Lean Software Strategies? I wish there were an equivalent version for other engineering disciplines. In fact I've started writing a few chapters of one.

I would rank "Toyota Production System," "Lean Software Strategies," and "Lean Thinking" as the best process books out there.

Blogger JCclimber May 26, 2015 10:02 AM  

I've been working full time in Lean Six Sigma for almost 10 years. Just finally got my certification this year because it is a HR resume screen that was blocking me from a number of job opportunities.

Mid-wit is absolutely correct. The most effective people I've seen using these tools are the average IQ people who do not get distracted and simply follow the rules and tools of the process (Six Sigma, not Agile), and the truly intelligent who know when to throw the rules out the window and just get 'er done.

The mid-wits get distracted and try to use every prescribed rule and tool even when they don't apply. Most of their projects fail. The average IQ individual asks one of the brilliant people for guidance and follow that guidance.

The mid-wits in my field are by FAR the most frustrating to deal with day in and day out. "Oh, I see you aren't using the important technique that *I* was taught in my class. Why is it that you believe you don't need to use that technique?"

Because I'm a frickin' master at this stuff and have successfully saved millions of $$$ for my companies, dumbass. The worst person to get landed on one of my project teams is one of these mid-wits who have some training in the field, for exactly this reason. I now just immediately plan to work around these people as soon as I identify them, or replace them if that is an option.

All my jobs in the field, by the way, have come from connections and none from blasting out resumes and following bureaucratic nightmare job application processes.

I looked into the Agile stuff, and I agree that it seems to be snake oil. Every one who has it is exactly as Vox described. I wasn't even tempted to get their certification when offered $90 per hour if I got that checkbox filled, because it wasn't worth working with bureaucracy on a constant basis.

Blogger maniacprovost May 26, 2015 10:02 AM  

...Although only LSS has practical advice.

Blogger Cail Corishev May 26, 2015 10:04 AM  

So the house is supposed to be done two weeks from now... and its all framed up and you're putting in the last bits of flooring and finishes... but hey... ya know... shouldn't the bathroom be over there?

Funny thing is, I have helped build a house sort of "organically" -- not to the extreme of adding new rooms at the last minute, but starting with a back-of-the-envelope sketch and figuring out the details as we went. It only worked because:

- It was a small house with a very simple design.
- The builders were also the owners and the residents, so they were the only ones stuck with changes they made, and they could continue making changes as long as they wanted.
- We did it in our spare time for fun and no one was getting paid, so it didn't matter that much if it went 200% over budget on time.

You can do small, hobby programming projects that way too, but you can't get away with that if you want to turn the work over to employees or contractors, or if you're up against deadlines. Try to build a house with Agile methods when you have different teams doing everything from foundations to plumbing to drywall, and it seems to me you'd have a lot of guys spending a lot of time having "meetings" under a shade tree.

Blogger JCclimber May 26, 2015 10:05 AM  

as a side note, I work primarily in Toyota Production System or Lean, rather than Six Sigma. Most American companies need Lean far more than they need Six Sigma.

Anonymous Stg58 / Animal Mother May 26, 2015 10:10 AM  

Small, focused teams are the best. In my industry, big companies don't innovate. Small companies like mine innovate breathtaking new process analytical equipment and big companies buy us. We them move on to the next sweet idea.

Blogger maniacprovost May 26, 2015 10:14 AM  

Sure, as long as the tech guys aren't being morons and running their "knee-jerk no" routine, which only makes them look like incompetent idiots

If I had a dollar for every time someone told me something was impossible... Actually, I do, because my job is to prove it's possible and then make the corporate engineering group do it. Usually it takes 6 months to wear them down. If we need something sooner, I have to do the work myself.

Blogger Eric Mueller May 26, 2015 10:15 AM  

I've always thought Agile sounds great on paper, but in my experience, when I hear a developer say he's going to use Agile, I think of three things:

1) No Planning
2) No documentation
3) No accountability

And a 4th, the project is going to fail and be cancelled. Usually after I've been put through the ringer because the developer can't keep a schedule.

Anonymous Jack Amok May 26, 2015 10:36 AM  

I could type half the day on this, but I have to get off to work for our Morning Scrum....

Bottom line, "Agile" is mostly a buzzword these days and you'd have a hard time getting developers to agree on what it actually is. But it started out as way to avoid massive overhead, including over-specified designs and "Architecture Astronauts" (people who designed architectures but never sullied themselves with the menial task of actually building them and running into all the problems).

But, like most buzzwords, the suits don't really know what they mean either, so use that to your advantage and tell them you're "doing Agile" so they get out of your way.

Anonymous Leonidas May 26, 2015 10:36 AM  

In order to finish my master's degree I had to sit through a "software engineering" course. A huge portion of the class was just a survey of different "process models" like "agile", "waterfall", etc. Most of the rest was talking about ways of breaking down software into easily tracked metrics.

Now... there's some degree of utility to all of this. Even the most highly motivated and capable team needs some kind of process to coordinate things, if there are more than about three people working together.

But these people are literally trying to break down software development into a "process" that a trained monkey could do: do step A, then step B, then C, etc and it will always work. The thing is, the idiots haven't realized that if they ever actually achieve their goal (which I'm dubious about, but let's play "let's pretend" for a moment) then they'd put all of us out of work. Once you manage to achieve that, you just program a computer to do it and there's no need for software engineers anymore. Poof, a whole field out of work.

The managers get this, which is why they want to do it (technically feasible or not). Think of all the money they could save in development costs! The "brilliant" tech guys who are striving to actually do it don't get it, and yet still want to pretend that they're the smartest people in the room.

For all that they pride themselves on their logic, I've found that technical folks aren't much better about applying it to their own behavior than the average teenage girl.

This. Exactly this.

Blogger Harsh May 26, 2015 10:41 AM  

#11 seems like a spectacularly bad idea.

Blogger maniacprovost May 26, 2015 10:50 AM  

Stg58, you work in downstream? Upstream, the smaller companies innovate more efficiently, but in terms of quantity more stuff gets engineered by the large companies.

Anonymous RS May 26, 2015 10:51 AM  

Deep Thought -

Probably because you've been involved in waterfall methodology projects that spent 3 years in development until their grand unveiling. The developers point to all the bullet lists of features they've created and how well it matches the reqs doc while the people actually making money for the company are pissed because while the software was being developed, business processes changed considerably and now the final deliverable is a half-working POS that has now chained moneymakers to some decrepit process. I've done data analysis, program management and strategic support at F10 companies all the way down to 50 person companies. IT folks routinely push their own solutions rather than listening to business users, pride themselves on 'finishing' projects rather than supporting the business, and fight/moan any time their ultimate super solution falls short because they confuse technical expertise with business knowledge and silo themselves inside the company.

Agile development isn't perfect, but it approximates the "get the smart people in the room with a goal and let them sort it out as fast as possible" much closer than other methodologies. No system can replace smart workers - you don't need Agile or anything else if you have a strong team of experienced developers.

Blogger Zach May 26, 2015 10:52 AM  

I've seen it work... once. Classic Kent Beck XP, pre-"Agile Manifesto".

One of the keys, if not the key, is the expected behavior of management for the process to work. I don't see this reflected in the Manifesto, which seems to be all about what developers need to do. If "Business people and developers must work together daily throughout the project" is understood to mean "We will schedule more meetings to berate the developers about why we are behind schedule until everything is better!", then it's not going to work.

Originally, XP had these twin principles:

1. Developers should not make business decisions.
2. Business should not make technical decisions.

So: Development team is responsible for knowing what they can do and presenting options. "Over the next three weeks, we can give you X or we can give you Y, but not both. We could get started on Z, but it won't be done by then."

Business team is responsible for understanding what creates business value and what is priority.

If the business team takes the input from the development team and says "OK, X is what we really need for the customer demo - do that and get as much of Y and Z in as time allows", then they are following the process.

If the business team takes the development team input and thunders down from on high "NO! FAILURE IS NOT AN OPTION! WE SHALL HAVE X + Y + Z, AND IT SHALL WORK FLAWLESSLY!!" - then they have broken the process, because they are attempting to decree what it technically possible over the judgement of the technical people.

In response, we know what is going to happen. The development team, knowing that X+Y+Z isn't going to happen, will make their own decisions based on their judgement of least technical risk, what's technically interesting, or their judgement of company priority. Now the process is not being followed again, because this is a business judgement.

Anonymous ZT May 26, 2015 11:03 AM  

I have worked with multiple "Agile" teams and I have seen it done well and done absolutely wrong.
I have worked with multiple "Waterfall" teams and seen it done well and absolutely wrong.

In Response to VD's Points

1: This is why you have a customer directly involved or a customer advocate. Take Apple for example. Jobs was considered the Customer Advocate. He drove features, requirements and said when things were other crap. Problem is many companies look directly to end customer (Which works if that customer is intelligent enough to work with) instead of an internal advocate. Using and Agile or Waterfall approach does not negate this.

2: Change does not negate cost. In fact Agile is more prone to communicate to customer that their request will have direct impact to cost of their request, time to deliver, or both. This is due to time boxed communication as opposed to milestone which is generally tied to when an item is ready. Agile also has Milestones.

3: Not sure you this would be CYA? This would be no different then the Alpha or Beta access in most games.

4: By talking they generally mean a 15 minute yes/no on progress. These should never be lengthy conversations. If it cannot be done in 15 minutes then they are doing it wrong.

5: No disagreement.

6: This does not negate the use of technology like Hangouts or video conferencing. Other than sprint planning periods (should only happen once every two weeks) and daily scrum (15 minute conversation, some even just do this as an email or notification in Jira or other Agile tracking software). If you have to talk the the project owner and customer advocate more than that then generally there is a problem in communication, either on their part or yours. Project methodology does not negate this. Some PM's are simply Stupid or having life crisis that are impacting work.

7: No disagreement

8: Problem with this is that it tends to think of people as robots. There is a problem of "took to f---ing long and left the project". Some people just get tired of not seeing progress. Look at Duke Nuke'm Forever.

9: This is referring to the feedback loop that is created. Because you are deploying and testing continuously, each new feature is rolled into a running system. Unlock waterfall where you often have a large mass of featured deployed at the same time, it makes bug fixing a pain in the ass.

10: I honestly do not see how your statement and 10th principle conflict.

11: I think this may be more of a misunderstanding of roles. All projects should have the role for the Project owner (Person who is ultimately accountable and provides vision for the team) and customer/customer advocate (Person who provides feed back on acceptability for use. This person may also contribute to the vision) to be present. The person with in a role may change but generally doesn't until completion of the project.

12: No disagreement

You might want to look at "Tearing Down the walls" at least in regards to Project management and customer advocacy. I have not read though it all though but it seems okay.

Also you might want to see Gene Kim's presentations

* DevOps Benchmarking (Numbers and benchmarks)
* DevOps/Agile Lesson learned (Kind of fluffy)

Anonymous Hoots May 26, 2015 11:04 AM  

Agile is NOT just a gimmick a la Six Sigma. In the large organization in which I work, it has led to drastic improvements is productivity and quality. Keep in mind I work in defense, which generally lags the industry by at least ten years, so maybe that's why no one else is impressed. Also in any large team you have some fat-of-the-curve engineers who benefit more from agile processes than others.

A couple points:

The system architecture is stable and well-communicated across a team of ~120 coders. System-level requirements are likewise stable, and only derived requirements are added/modified via agile processes. Basically, the foundation and framing are in place.

The main purpose of constantly delivering working product is to quickly identify and resolve design issues. Problems that might have taken a year to discover in the past now might be fixed in a month, and at far less cost.

The best engineers force the process to serve the team goals rather than allowing the work to serve the process.

Anonymous Stg58 / Animal Mother May 26, 2015 11:06 AM  

Maniac,

I work in every segment. Analyzer manufacturers are only classified by where their equipment can be installed. Our analyzer can be installed from the wellhead to the loading terminal. As a matter of fact, we are the only company who can perform crude oil composition and properties analysis at the wellhead, pipe or truck in real time.

We are literally creating a market out of thin air. The small companies like mine innovate the actual analysis equipment, and are bought by larger automation and analytical companies, such as Yokogawa buying ASI, or Emerson buying Cascade Technologies.

Blogger FALPhil May 26, 2015 11:15 AM  

Vox said:
You mean milestones? Which we were using long before Agile ever existed? That's literally how I've always done software development. What I think is you've just convinced me that Agile is nothing but marketing.


I have seen Agile work on several occaisions. However, all of those occaision have had 2 common factors:
1. There was a dedicate requirements team that was not the development team.
2. The development team worked the backlog, and successful testing was the measure and not the milestones; each drop of deliverables was a cohesive testable unit.

The key was making the backlog a higher priority than the calendar.

Stg58 / Animal Mother said:
Small, focused teams are the best. In my industry, big companies don't innovate. Small companies like mine innovate breathtaking new process analytical equipment and big companies buy us. We them move on to the next sweet idea.


We should all remember this as a truism. In my 25 years in the business, I have never seen anything more consistent. Small companies empower people and don't have the middle management bureaucracy.

Anonymous SumDood May 26, 2015 11:16 AM  

Six Sigma is for manufacturing environments. Yet many people try to apply it outside of that environment. Does not compute!

Blogger FALPhil May 26, 2015 11:17 AM  

Vox said:
You mean milestones? Which we were using long before Agile ever existed? That's literally how I've always done software development. What I think is you've just convinced me that Agile is nothing but marketing.


I have seen Agile work on several occaisions. However, all of those occaision have had 2 common factors:
1. There was a dedicate requirements team that was not the development team.
2. The development team worked the backlog, and successful testing was the measure and not the milestones; each drop of deliverables was a cohesive testable unit.

The key was making the backlog a higher priority than the calendar.

Stg58 / Animal Mother said:
Small, focused teams are the best. In my industry, big companies don't innovate. Small companies like mine innovate breathtaking new process analytical equipment and big companies buy us. We them move on to the next sweet idea.


We should all remember this as a truism. In my 25 years in the business, I have never seen anything more consistent. Small companies empower people and don't have the middle management bureaucracy.

Anonymous hsu May 26, 2015 11:21 AM  

As someone who does agile everyday, the most useful part is the 2 week sprints.

The 2 week sprints forces more realistic estimates from developers and more realistic goals from management. You cannot design and build the world in 2 weeks, and everyone knows it, so everyone takes steps to breakout what could be done in 2 weeks.

When you get to 6 month cycles, instead of 2 week cycles, management always tries to add more features than possible, and developers woefully underestimate their time. Basically, all but a few folks end up vastly overestimating their abilities.

That's the main reason I'm a proponent of agile, even though I hate all the meetings and overhead it produces.

A group of people with good feature/time estimation skills, agile becomes a detriment, but having such a group of people is as rare as a unicorn in the software industry.

Blogger Danby May 26, 2015 11:27 AM  

That list looks like the recipe followed by every massively failed project I've ever participated in.

For example: You might start using a NoSQL database just as a fast key-value store. However, you keep pushing on that piece, you wind up storing things that start to look relational, and suddenly you're writing business logic controls to do what a RDBMS does for free out of the box, and you've lost the idea of your fast key-value store altogether, because you let things grow organically and satisfied the objective for the next meeting, not the big picture.

I worked on that project. Literally, that was the failure mode for a $30M project, where the development manager just couldn't say "no". Or even tell the truth for than matter. And refused to pay for licenses for an RDBMS. So we wound up using an open source one with a few critical stability problems, in a telephone system. It all ended with corporate collapse and bounced paychecks. I was still young then and thought I could ride that tiger.

In response, we know what is going to happen. The development team, knowing that X+Y+Z isn't going to happen, will make their own decisions based on their judgement of least technical risk, what's technically interesting, or their judgement of company priority.

Actually, if they've been in the development world long, they're getting their resumes together and talking to their friends and acquaintances to see what openings are available. There's a certain smell to a doomed project. Once you've ridden one down into bankruptcy you learn to avoid it.

I'm not a developer myself, but software development isn't fundamental research in mathematics or physics. The process is fairly well understood. The problem is that good practices require people to leave a little of their egos behind.

One project I worked on, we had a new manager. The base of the project was a re-write of a software product that had gotten crufty and unstable with age. Everyone tried to make it clear to him that within our context, most of the work was going to be defining data structures, interfaces, etc. We knew how it worked in the guts, and we might even be able to re--use much of our code, but we needed to maintain and support it, so it needed a re-write.
So, after a week of one-on-one meetings about the project with literally everybody in the department, who all told him pretty much the same thing, he presented the project development schedule, which was front-loaded with specification and definition work and actually had coding start about a month in.
All of which lasted exactly one week. Within a week he was knocking doors looking for code.
I left the group before the 1st reset. The original schedule was a 3 month project. After 4 months they threw out the code they had and started over. And again 6 months after that. They were sold to another company for their legacy software and patents a couple of months after that.

Anonymous Stg58 / Animal Mother May 26, 2015 11:27 AM  

Zach,

Sales and marketing in our industry has to make the decisions on what applications, markets and analyses we will chase, in coordination with the technical/R&D/Engineering side. If nit, the factory will just continue to make what they always make, and will be blind to market developments that necessitate the engineers from taking more risks than they want to. Sales and marketing has to push the factory. We're the picket lines that the customers hit, otherwise there would be a disconnect between what the customers want/need and the factory makes.

Anonymous Huckleberry (#87) -- Certified Scrum Master May 26, 2015 11:40 AM  

The only true benefit from an agile/scrum approach to development is that showstopping bugs and errors are usually discovered much more quickly.
It's big issue is that it's a time sink.
Too many productive hours are spent servicing the process rather than actually producing anything,
But more than anything else, it's a goddamn cult.
I wasted 2 days at a seminar earlier this year focused on scrum/agile and I was just waiting for someone to break out the Kool Aid.

Blogger VD May 26, 2015 11:40 AM  

Programmers can estimate if something is an hour, a day, or maybe two days. Beyond this, programmers are incapable of estimation.

I learned very early on that "two weeks" is programmer-speak for "I have no idea how long it will take".

Blogger RobertT May 26, 2015 11:43 AM  

The worst offender of all was Management by Objectives. You create the objectives for your department and you're evaluated by whether you meet them. Worked under that for a while. Complete idiocy. But even more idiotic ... I recently learned this is the framework around which Google's management philosophy is constructed. Probably the influence of the Montessori educations of both founders. This just shows you that money raining from heaven causes more people to think they know what they're doing than anything else in the world.

Blogger SirHamster (#201) May 26, 2015 11:46 AM  

Agile is also used as a catchall term for vaguely defined, hodge-podge proceses. My own experience at a new job:

"What's our team's process for software development and release?"

"Uh.... we're Agile."

(no further elaboration)

My translation: Nothing.

Anonymous patrick kelly May 26, 2015 11:47 AM  

We call it hangin' on the bleedin' edge.....

Anonymous dh May 26, 2015 11:53 AM  

There's a certain smell to a doomed project. Once you've ridden one down into bankruptcy you learn to avoid it.

Yes, this a thousand times. By the time management usually figures it out all the good ones are already ready to go on to a better project. And all you have left are the dead enders.

Blogger BC May 26, 2015 11:54 AM  

Curious to see if any of the Ilk have read Richard Sheridan's Joy Inc. He tells the story of creating his own software development firm which seems to have avoided many of these pitfalls by cutting out the middle layers of management, keeping everyone accountable at all levels, involving the customer in the specific requirements and changes thereto, and testing each individual component as they go.

Blogger Aeoli Pera May 26, 2015 11:54 AM  

There is only one way to write software:

1. Put an AS/HFA in front of a computer.
2. Wait.

Where's my Gates Foundation grant?

Anonymous OffTheCuff May 26, 2015 11:57 AM  

If you like agile, it's because you're a shitty manager or an average developer.

I worked -- past tense -- on a highly functional team of very smart folks for 14 years. When the company was purchased, "Agile" was adopted and they circulated the manifesto, I though to myself "yeah, we already do all of that" and ignored it.

Then they forced the horror if Scrum on us, top-down. Suddenly, a team that used to produce excellent results was mired in meetings all the time, and became design-by-committee. Progress ground to a halt, though the illusion of progress skyrocketed.

If you're a good developer, and anyone above you breathes a word that even rhymes with "agile" or "scrum", quit immediately. Don't wait for your project and soul to die.

Anonymous patrick kelly May 26, 2015 11:58 AM  

My experience with those at the top who promote "self" organizing or managing teams is they don't want to be accountable for any decisions they make or authority they exercise over the team, and they will....all the while insisting they're not interfering, or micro-managing, they just want to empower the team to be successful....blech....and a rare few really understand what this means and try to do it.....the other's can all foad afaic....

Either trust and invest in the team you've put together, or have the courage kick or fire asses until you have one you trust and believe in.......
/rant...

Blogger Chiva May 26, 2015 11:58 AM  

"Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done."

I had a good laugh with this one. When done correctly this usually results in a very good product (participated in 2 projects like this). The issue is that most SW management teams cannot allow themselves to trust their most motivated employees to get the job 'done'. It is a rare SW management team to take second seat and actually support their employees.

Blogger Danby May 26, 2015 12:03 PM  

11. The best architectures, requirements, and designs emerge from self-organizing teams.

"And for today's magic trick, we're going to organize a self-organizing team!"

I mean, While obviously true, one could also say that for example, self-managed employees have the best time management.
Knowing that little (almost) tautology does you no good at all.

Anonymous patrick kelly May 26, 2015 12:03 PM  

" It is a rare SW management team to take second seat and actually support their employees."

Heh, a kinder, gentler way of putting it.

I despise being wrapped up in the blame game....just f'n figure out what's wrong and how to fix it........decide who to move or replace later....
...I've often had to fess' up to messing up, being wrong, lazy, or stupid and express the above.....I'm amazed it has not resulted in me being canned...yet......

Anonymous Sun Xhu May 26, 2015 12:07 PM  

ITIL is similar, from the IT support area, and just as idiotic.

"Best practices" written by some guy at Microsoft, Oracle, or Google, who have never actually worked at a real company, with limited resources and drastically variable priorities.

Blogger Feather Blade May 26, 2015 12:10 PM  

I learned very early on that "two weeks" is programmer-speak for "I have no idea how long it will take".

Boy howdy is this true. My unit was assured that the ITS department would have one of our online forms up in two weeks.

...Nine months later, we're still waiting.

Anonymous Nathan May 26, 2015 12:11 PM  

If I remember correctly from my ITIL class, I believe that it was spawned by the British government. At best, it seemed a strange mix of not-quite-complete-bullshit-but-pretty-damn-close. However, I'm also saying this as one of the guys most of you don't want on your team...

Blogger Aeoli Pera May 26, 2015 12:12 PM  

Oh! I have another one. How to fix common techie personality problems:

1. Make them lift heavy weights regularly.
2. Require gains.

That's it.

Blogger Brad Andrews May 26, 2015 12:21 PM  

You have to view it in the context as the significant alternative to the traditional waterfall approach. (Or a complete lack of any structure.)

Some ramblings (long because I came late to the party):

- The daily meetings are supposed to be brief. Dragging them out means you are not doing it right. While all may not stand up during them, they are called stand up meetings for a reason.

- The aimpoint of many here is likely biased because of the focus on a fixed final project. The books CH produces, the games VD has made in the past, etc. all had a fixed deliverable. It may have been enhanced or patched, but a certain final goal was the target with limited follow up updates.

First Sword will likely test that methodology more as it will need constant adjustment if it is going to continuously generate income. An iterative approach is more likely there.

- I would argue this is somewhat like the difference between writing a novel with the approach of bullet pointing everything out and that of having a general goal, perhaps with a few key elements, but adjusting as things go along. I think VD took the latter with AToB IIRC and it worked well enough, but GRRM is (per the comments here as I have not read any of it) doing the same with his series to the detriment of the series.

How many of you have delivered software that is used by others in a retail setting on an ongoing basis? That makes a strong argument against the long term waterfall approach.

- Everyone cannot be above average, by definition, so you have to work with what you have. Only a small number of companies can always hire the best. That means that an approach that applies to what is available is often needed rather than one that relies on having perfect people.

(A side note: It strikes me as humorous that many organizations want to higher better than average people with industry average pay, but that is another argument.)

- I do question whether the purely iterative approach can really scale well to really large systems, but the old waterfall approach has not done that well either, so the alternatives are not great.

- Software development is not easy any more than carpentry is easy. Yes, the basic tasks are known and can be replicated, but I am never going to look like the guy on New Yankee Workshop, no matter what structure I use. I can sling code if I ever get back into that. The reverse is also true in my experience teaching coding and working in the field for many years.

- I would rather have a smart pilot flying the plane than a dumb one, but I still want him to use a checklist. I don't trust his smarts that much.

- The idea that an end plan is all that is needed is idiotic. I guess none of you promulgating it have ever gotten lost while performing a clear, simple well defined task on the Internet, or even Amazon. Anyone read Wikipedia and ended up on a completely unrelated topic 30 minutes later?

Creative minds are just that and can stray, some solid structure is quite helpful for reigning in the squirrel chasing.

(cont)

Blogger Brad Andrews May 26, 2015 12:21 PM  

- Complexity points are supposed to indicate difficulty of implementation. What other complexity would they cover?

- Those who seek certification on something are not necessarily wrong, but doing so just for the certification is a waste of time, you need the underlying skill.

You can find the same in the information security field. I have a few certifications because I decided to get them, not because they made me better in and of themselves. I use them as needed for some audiences, but I rely on my experience.

The same would be true of agile approaches. I would not hire a company merely because it used agile approaches, but I would like the fact they used many of its features as compared to the alternatives.

This seems like another area that many are applying black and white viewpoints when it is not quite so clear.

TL;DR - It is a better approach overall than the waterfall alternative. Development is not as simple as some claim though, so any methodology will have challenges. Management will be management.

Anonymous patrick kelly May 26, 2015 12:22 PM  

"I learned very early on that "two weeks" is programmer-speak for "I have no idea how long it will take"."

That's a result of tiring from trying to honestly express my professional opinion that there is no way I could provide a reasonably accurate time frame for what they are asking. I have yet to discover the magic words to successfully communicate this in a way the person asking will understand, hence adopting the "two-weeks" shibboleth ......

Anonymous Isidore The Farmer May 26, 2015 12:23 PM  

I participated on an agile project for a large system replacement project ($20-25 million) in a mid-sized corporate environment, involving lots of outside consulting resources from one of the bigger consulting / accounting firms. This project used lots of overseas resources, and I believe the agile methodology was used, primarily, to keep close tabs on that group. Frankly, they had to be closely managed, so some of the agile principles made sense in that context. The language barrier was immense. It really was a lot of meetings, a lot of meetings planning for the meetings, etc. That ate up any savings you supposedly get by focusing less on documentation. And, frankly, if I had to choose I'd prefer the documentation, given business environments that often keep applications around for 15-20 years (sometimes more). But there was no choice but to manage the group that tightly, given the constraints of the project (money, time, available implementation partners).

Most of these processes are developed in response to the fact that the tools have become more complex over the last 25 years, and there is simply a smaller pool of people who can do it well. A decent to slightly above average COBOL programmer from 25 years ago (of which there were lots) would not be a good developer at all today because the tools and technology have become much more complex. If he was still a good programmer, it is likely because he specialized in a very tight area. But, this specialization makes him less useful to most companies who need broader skills.

None of these methodologies, in my opinion, are designed to serve entrepreneurs already working in small groups with known goals in mind. These are for mid-size to larger companies that struggle finding enough competent people to meet the demand of the work needed, and also find themselves chasing shiny objects for some new competitive angle. It will always be an uphill battle in that environment because there just aren't enough people with an IQ above 125 to fill the empty seats.

I currently work for a place that outsources the management of the company's network / server infrastructure to a service firm specializing in this. They screw up all the time. They're good guys, but really smart people don't understand how complex all of this has become for people in the 100-125 IQ range. I'm at about 125, and it taxes me much more than it did 15 years ago. We're a mid-sized company with locations throughout the company, and I'm responsible for managing the IT function: phones (IP-based), servers, network, business applications, desktop management (thick/thin clients), printing, etc. I'll be the first to admit I'm near or at my limit. If I hire people with an IQ of less than 120, they'll only be able to learn a small segment of everything we support. And finding people at 130+ is extremely hard to do.

I would bring the server/network/support functions back in house....but I would struggle finding truly competent people even more than the local provider who specializes in it. The truth is, the tools have just become more complex than most people can handle. Any time you have a function that requires a 130+ IQ to be genuinely competent you are going to have problems. And realize, most people aren't designing

All of these fancy processes are simply people trying to compensate for the nature of the IQ bell curve. But you can't compensate for it...not very well, anyway.

I am definitely in the midwit range, so I could be wildly wrong...

Blogger Brad Andrews May 26, 2015 12:24 PM  

I would ask a general question: How would you structure development in a middle to large size company?

Not just the on off project, but an ongoing effort for ongoing use by others, including regular enhancements.

Anonymous patrick kelly May 26, 2015 12:32 PM  

I'm not an alpha type, so I like strong, decisive, smart leadership to start with. Anything else makes me nervous........and I want to know they have my back....either to protect it, or kick me in the rear when I need it....

Blogger VD May 26, 2015 12:38 PM  

That's a result of tiring from trying to honestly express my professional opinion that there is no way I could provide a reasonably accurate time frame for what they are asking. I have yet to discover the magic words to successfully communicate this in a way the person asking will understand, hence adopting the "two-weeks" shibboleth ......

Why do you find it hard to tell them "I don't know"? I mean, you do realize that within two weeks they're going to find out that you don't know what you're talking about, right?

Anonymous Stg58 / Animal Mother May 26, 2015 12:41 PM  

The word NO is magical....

Anonymous physics geek May 26, 2015 12:42 PM  

I worked as a software developer/data analyst on two different Agile teams. No one, and I mean no one, spouted any of this gibberish. Instead, we divided and conquered the two (rather large) tasks at hand. Oddly, a group of skilled and dedicated workers were able to determine a solution to a new problem. Or, and I'm just spitballing here, maybe it wasn't so odd.

Blogger Danby May 26, 2015 12:44 PM  

Part of the problem here is that development is not one thing. A web page for access to a catalog is very very different from an embedded service kiosk, is very different from a real-time enterprise help desk. Each requires a different approach.

The principles of Agile lead me to think that it would work well for a specific class of problems, well defined with lots of libraries to handle the detail work and standards or at least defined interfaces and data structures. For other classes of problems, it would not work well.

Anonymous RedJack #22 May 26, 2015 12:45 PM  

Six Sigma, Lean, and other programs are tools.

We are going through a Lean process at my plant now, and I have learned to appreciate them for what they are. They are a process to help those who need, they shouldn't be a shackle for those that don't. Unfortunatly, they are often used that way (Same as ISO).

For the guys on the floor, it is golden. For telling the people above me what I want them to here, the same. For doing my job, not so much.

Blogger Lew Rand May 26, 2015 1:07 PM  

Wrote my bullet points before reading VDs (but mine line up pretty well)

1) Rapid Prototyping. Been delivering functional prototypes for decades
2) The customer has no clue what he really wants. Thus the need for 1
3) See 1
4) Can I just say "Smirk". More 2 than anything
5) Check
6) Actually no. Face to face is hard to schedule, and quite often a good developer actually UNDERSTANDS the client and can fix his problems without wasting everyone's time. Also stolen from the comments, write it down and mail it to me or this conversation didn't happen. (Usually forgotten unless its an obvious BIG problem)
7) See 1
8) Yeah that always works. Not.
9) Yeah. Great artists do great work. Team of average artists might be able to paint the rooms with only some paint on the trim.
10) Check
11) No, the best ones really do come from one or two people who know best. But they usually need to be hands on people too who have to implement/work with the choices.
12) Rabbits can self reflect? (As you can tell I'm not the best team player)

Also, I don't even remember the last time I had to say 'two weeks'. Any project that takes longer than 2-3 days probably needs to be 'refactored' into smaller projects/tasks so the work doesn't go all fuzzy while completing it.

Oops, I did say two weeks when we had a company at then end of a contract that all of a sudden wanted definitive time schedules for vague items. Lets say i wasn't highly motivated to work hard for them (especially when they tried to bankrupt us by refusing to settle their debts and figured we wouldn't survive it).

Anonymous patrick kelly May 26, 2015 1:16 PM  

Vox: "Why do you find it hard to tell them "I don't know"? I mean, you do realize that within two weeks they're going to find out that you don't know what you're talking about, right?"

It's not a difficult mystery to either party, but a dance to a song we both recognize may never end....I don't find it difficult at all to tell them, they just usually won't accept the answer, even if I suggest someone else might have more information or insight....I'm a bit of a paranoid peon...I really think the sales/marketing critter really just wants someone to tell them what a customer wants to hear, then tell the customer, and when it doesn't happen, blame the poor programmer..........

Crap, my attitude is just not all that great lately, I really should at least attempt a better answer, like "If appropriate resources can be re-directed to this effort, I think two weeks is do-able for a first proof of concept version..we need to discuss this with other people involved..." or something like that, but I'm often faced with, "I need to know what to tell the customer right now..." etc, which may give you a glimpse into how well things are run here over-all, and yet we keep producing and growing.......

If someone in authority over me asks I'm usually a bit quicker to provide the most accurate, honest answer I can and let the chips fall where they may.....weak dissembling has worse consequences later....I'd rather get the pain over with now.....

This doesn't really happen much lately...just the first few of years I was low man on the team.....current status affords me the luxury of getting to tell sales/marketing their idea will never make it into the current product, please make your suggestion for the next version to the appropriate person ...who is not me.....if it is someone in charge I point out there are other people who should be included in any serious discussion about it.....any change has consequence, someone has to decide if it is worthwhile.......

I'm venting old wounds here.....sorry.....thanks for the therapy session........and thanks for asking me a serious question, helps me evaluate my place in the world.....tmi eh?

Blogger Danby May 26, 2015 1:21 PM  

"I need to know what to tell the customer right now..."

"Tell him you're an idiot and you have no idea what it takes to include his new requirements."

Anonymous AlteredFate May 26, 2015 1:22 PM  

Why do you find it hard to tell them "I don't know"? I mean, you do realize that within two weeks they're going to find out that you don't know what you're talking about, right?

Under promise, over deliver. Most people have that precisely backwards. Telling your customer exactly what they want to hear might sound great, but when they discover you are full of shit, it's hard to bounce back from that.

Blogger Brad Andrews May 26, 2015 1:24 PM  

Talking about estimation and getting things done: I learned in a past position it was better to pad time than to get things done too early. I had guess something like "2 weeks" for a new feature and ended up coding it in about 2-3 days. I had not thought that possible, but it played out well. I was chastised for not estimating well rather than complemented for doing it quickly.

I never have mastered estimation for some software development things, but I agree that many things really are a stab in the dark.

Agile does attempt to break it down into smaller pieces, at least in theory.

Blogger Stilicho #0066 May 26, 2015 1:26 PM  

I really think the sales/marketing critter really just wants someone to tell them what a customer wants to hear, then tell the customer, and when it doesn't happen, blame the poor programmer........

Ah, so you're saying he has people skills?

Blogger Danby May 26, 2015 1:28 PM  

I had guess something like "2 weeks" for a new feature and ended up coding it in about 2-3 days. I had not thought that possible, but it played out well. I was chastised for not estimating well rather than complemented for doing it quickly.

Perfect case for screwing around for 5 days and delivering 2-3 days early.

Anonymous clk May 26, 2015 1:30 PM  

Kirk: How much refit time before we can take her out again?
Scotty: Eight weeks, sir. But ye don't have eight weeks, so I'll do it for ye in two.
Kirk: Mr. Scott. Have you always multiplied your repair estimates by a factor of four?
Scotty: Certainly, sir. How else can I keep my reputation as a miracle worker?
Kirk: [over the intercom] Your reputation is secure, Scotty.

Blogger rycamor May 26, 2015 1:32 PM  

Zed Shaw had a most eloquent response to Agile.

Anonymous patrick kelly May 26, 2015 1:39 PM  

"Perfect case for screwing around for 5 days and delivering 2-3 days early."

I don't get to screw around more, I get to work on the other code that was "the most importantest thing in the universe ever..." just two days before.......

Blogger Zach May 26, 2015 1:54 PM  

VD,

Yes, but you're willing to take "I don't know" as an answer. Lots of organizations are not.

If management decrees that "all tasks and bug fixes SHALL have estimates!" ... and provided negative feedback for "I don't know / no estimate" ... but doesn't seem to care so much about "two weeks" actually taking three or four or more...

... then they are going to generate a lot of "two weeks" estimates.

Anonymous patrick kelly May 26, 2015 1:57 PM  

We don't really have any management, just interrupt driven panic....

Blogger Chiva May 26, 2015 2:05 PM  

We don't really have any management, just interrupt driven panic....

A project does move into the surreal when management operates in that mode.

Anonymous Orville May 26, 2015 2:11 PM  

- I do question whether the purely iterative approach can really scale well to really large systems, but the old waterfall approach has not done that well either, so the alternatives are not great.

Musk's SpaceX seems to be pulling this off successfully on a large project. In terms of space hardware instead of software it seems to me that SpaceX uses an agile like approach and ULA uses waterfall. SpaceX is already flying and making incremental changes to arrive at a manned capsule, while ULA will not be flying any test objects until 4 months to the end of the development period.

Blogger bob k. mando May 26, 2015 2:18 PM  

RS May 26, 2015 7:09 AM
Changing reqs doesn't mean scope creep



fuck you, pay me.

https://youtu.be/6h3RJhoqgK8?t=12m30s

Blogger maniacprovost May 26, 2015 2:23 PM  

Software for business processes is usually garbage because the business processes are not defined or understood by the people carrying them out, much less the programmer who eventually tries to implement them.

Development is outsourced, and there is no sustaining effort.

Result: A set-in-stone piece of software that precisely replicates the process used at one point in time by one department, while completely failing to capture the underlying concepts that could have made the software flexible, efficient and useful for many years.

I say this as a user.

Blogger Danby May 26, 2015 3:06 PM  

Iterative development assumes that you haven't made any mistakes in defining the problem, that any corrections will be straightforward. If you have a complex problem and happen to pick an inappropriate approach (more likely if you're Agile(tm)), or the wrong tool, you WILL fail. Iterative development simply ensures that will fail as quickly and as publicly as possible.

Blogger Danby May 26, 2015 3:07 PM  

@Bob K. Mando
uck you, pay me.

yes, this.

Blogger Markku May 26, 2015 3:10 PM  

Why do you find it hard to tell them "I don't know"? I mean, you do realize that within two weeks they're going to find out that you don't know what you're talking about, right?

In two weeks they will have entirely changed what projects they want and on what priority, so no, they won't. The estimate is obsolete by then.

I too have adopted this almost instinctively, having found that "I don't know" is just not accepted by management. Ever. So, giving them some number makes them happy and I get to go back to work. But it is indeed so that 90% of the time the truth is "I haven't the foggiest"

Blogger Markku May 26, 2015 3:22 PM  

My heuristic is that if I hear a feature, and am able to immediately understand it, then it's probably a twoweeks-ish feature. If the estimate is going to be wrong, then the reason is usually that the boring monotonous footwork -part of it was bigger than I thought.

It's usually immediately obvious what the difficult problems are, when you hear the feature. The amount of boring footwork is MUCH less obvious. That constitutes the unknown factor between the estimate, and how long it turns out to take.

Anonymous Banjo May 26, 2015 3:30 PM  

I think part of the appeal of Agile for companies that have... less than stellar personnell... is that they can churn out software by the boatloads which can act as an anchor around the customer's neck.

Sprint 1: Delivered X
Sprint 2: Fixed X and added Y
Sprint 3: Re-fixed X and fixed Y and added Z

etc

By the end of the project, a lot of code was delivered and some things are working, but now the hours are gone and these last few features can't be added without a major refactoring (and maybe re-write) because the design was half-assed at the beginning and wasn't good enough. But hey, the customer has already dumped all this money and they do have a "working" product. That should be good enough right?

Shades of Clark Griswold driving off with the Wagon Family Truckster...

It's also definitely marketing driven. We lost a bid on some services work that had NO TECHNICAL COMPONENT. We since discovered that the winner proclaimed their methodology used the "Agile approach used in software development" so they could more quickly and cheaply map business processes and because "...requirements cannot be accurately articulated up front".

So basically, they designed the system they way they wanted with input here and there from the customer. Also known as fling-shit-at-the-wall-and-see-what-sticks approach.

We took over during Phase 2 of this project and it's like Phase 1 never happened. That group essentially stole money.

Anonymous Banjo May 26, 2015 3:30 PM  

Whoops, sorry for the wall O' text

Anonymous patrick kelly May 26, 2015 3:41 PM  

Wow, I guess I'm not the special freak I thought I was....I luv this blog......the evil aliens from Uranus are at war with all programmers, not just me.....

Blogger Markku May 26, 2015 3:47 PM  

It is kind of the housetraining of the software engineer. Fresh in the industry, he will say "I don't know" where true. That goes over VERY badly with management, because everybody else is giving them bullshit numbers, and they have ONE job at that meeting: Extract a number from the engineer. Everyone else can do it, why can't you?

So, eventually you learn that the only way to get back to actual work is to provide a bullshit number. If it turns out bullshit, what are they going to do, fire you? Everybody else's numbers are bullshit too.

And since the project calendar is going to be so completely reshuffled by then, in the overwhelming majority of cases it cannot be revealed as bullshit even in theory.

It is a dysfunction of the field that's probably impossible to fix, so the dance of the bullshit numbers will go on forever. Every time you do it, a newbie engineer somewhere feels more pressure to do it.

Blogger Markku May 26, 2015 3:50 PM  

But I like it I'm not gonna crack

Blogger aut0x3ematthew May 26, 2015 3:53 PM  

If a team can use distributed version control, it can use distributed communications.

Sprint reviews are harmful to morale; anything worth reviewing has already been shown to those who matter in the course of implementation.

Blogger Ursus Maritimus Australialis May 26, 2015 3:56 PM  

Heh. This reminds me of my days as a developer, back when Total Quality Management was all the rage. They had a project management consultant come in to present a two-day class to all the developers.

The guy spouted things like "Time estimates should be made with 50% certainty." Uh, yeah. Would you please tell that to my management? Around here, time estimates have to be made with 100% certainty...

At least that environment taught me two very important things: 1) under-promise and over-deliver, and 2) document EVERYTHING. If we go over estimate because of scope creep, I need to be able to prove that they changed the requirements at the last minute.

Hrm. How did the Programmer's Night Before Christmas end?...

Compilers compiled, the programs computed,
The user's last changes were even included.
And the user exclaimed with a snarl and a taunt,
"It's just what I asked for, but not what I want!


(VFM 0xFF, Lowest Ranking Minion of the First Byte)

Blogger aut0x3ematthew May 26, 2015 3:59 PM  

"It is kind of the housetraining of the software engineer. Fresh in the industry, he will say "I don't know" where true. That goes over VERY badly with management, because everybody else is giving them bullshit numbers, and they have ONE job at that meeting: Extract a number from the engineer. Everyone else can do it, why can't you?"

Yes. I black-knighted at one gig by making meetings run long over discussions of how exactly to use story points in Jira.

If you're a known good dev, you can game this by seriously under-promising, then over-delivering unassigned things that actually matter, or paying off technical debt.

Blogger Cail Corishev May 26, 2015 3:59 PM  

Any project that takes longer than 2-3 days probably needs to be 'refactored' into smaller projects/tasks so the work doesn't go all fuzzy while completing it.

Yep. Vox is right, "two weeks" means "I have no idea," except I would add, "but two weeks is safe because I can't imagine this taking nearly that long." When that's become a problem was when I didn't know how long it would be because I didn't know the requirements nearly well enough, and sometimes because it took 2 weeks of back and forth just to help the client figure out he requirements before much real development could start.

So "two weeks" is a sign that I don't know the requirements yet, and breaking it down into smaller chunks will either reveal those requirements or show where they are missing and need to be filled in.

Blogger automatth0x3ew May 26, 2015 4:03 PM  

"Any project that takes longer than 2-3 days probably needs to be 'refactored' into smaller projects/tasks so the work doesn't go all fuzzy while completing it."

Agreed. Modularity, separation of concerns, semi-stable interfaces at every level: these make small projects possible and often obvious.

Blogger Danby May 26, 2015 4:18 PM  

Scott Adams wants to remind you of the other reason for the "two weeks" time estimate

Blogger Markku May 26, 2015 4:47 PM  

I wouldn't be TOO harsh to the agile team 1 whose product was crap, and had to be re-done by team 2. What team 1 accomplished, was to pin down the customer requirements down. No small feat. Once you know them right from the start, your job is immensely easier.

My university has a course where during one semester, you have to do a real-life software project for a real company or entity, as a team of students. In my year, there was one team that worked for a pharmaceutical company. They were not able to deliver any code at all. Normally this would mean F, no questions asked. Yet, the client was VERY happy, said the professor. Why? Because in the course of documenting all the medical jungle, they went from understanding jack shit, to understanding exactly what they wanted from the software. True, they didn't yet have any, but they had a document that they could give to the next team or company, to purchase the project. So, the team passed.

Blogger Markku May 26, 2015 5:11 PM  

As for Agile, it is a great thing for what it is: A reaction to the waterfall model. The points are not meant to make sense in isolation, but in contrast to waterfall.

Blogger astrodominant May 26, 2015 5:14 PM  

Agile, Lean, Six Sigma,TQM, etc. are just different tools in the toolbox. I work for an oilfield equipment manufacturer and we use each of them as needed to solve problems. We just pick the best tool for the job. It is not part of our day to day operations.

We also business process map any business that is going through any software implementation or improvement activity. This gets the business and the developers closer to being on the same page.

I am project champion on a 66 site, 22 plant, 6 country ERP implementation. We are replacing 17 ERPs with one. I spend just as much time pounding on the developers about scope creep as I do the business. Everyone wants to add something or has a great idea. On easily two thirds of our site kickoff/discovery sessions I am reigning in the implementation team to stop over promising things we can do but are out of scope.

Blogger Danby May 26, 2015 5:34 PM  

I spend just as much time pounding on the developers about scope creep as I do the business. Everyone wants to add something or has a great idea.

Or just things they are used to. Nobody ever wants to give up a feature or a function, even if they use it once a quarter.

Anonymous rho May 26, 2015 6:04 PM  

Agile, XP, whatever the latest project management fad is all boils down to the confused state software development is in today. Web development especially is more like herding wet cats through 7-dimensional space while some asshole aims a laser pointer at a disco ball.

Project management is actually very hard, and there are very few people who are any good at it. The temptation is to promote a wizened developer (average age, 27 1/2) to manage a development team of eager beavers (average age, zygote), but programmers rarely make good managers. If they're good, they can do it on their own and won't know what to do with their team; if they're not good, they can't see the forest for the trees and will send minions out to build a house at the bottom of a volcano.

Non-programmers can be even worse. They can't tell the difference between a good programmer and a great programmer. A great programmer isn't twice as good as a good programmer. A great programmer is an order of magnitude better. You can't lash a good programmer and a great programmer together and get a 20x multiplier. Nor can you add more developers to a project to speed it along any more than you can add another mother and get a baby in 4 1/2 months.

I'm quite sure that Agile works, but so can literally any other organizational and procedural model that doesn't involve gladiatorial combat or ritual sodomy. Hell, until 3:00PM yesterday afternoon the Linux kernel was maintained by Linus receiving patches via smoke signals and semaphore. It worked, because Linus is very smart, the kernel core organization was essentially flat, and the project focus was by definition finite.

If Agile, or XP, or Sex Ninja Coding or whatever hammers the project into something consumable by a development team, then it's probably going to work okay.

(I'm off to develop the 13 Crystalline Rules of Sex Ninja Coding.)

Blogger rycamor May 26, 2015 6:57 PM  

astrodominant May 26, 2015 5:14 PM

I spend just as much time pounding on the developers about scope creep as I do the business.


Can't tell you how many times I've come across a situation where some programmer inserted his insanely complicated pet idea into a project, to absolutely no one's benefit. Just recently I was looking at a registration form for a CMS that generated literally 140 SQL queries just to draw the form. The developer had come up with an exquisitely-crafted system to decompose every form field, every label, every possible set of values, every form group, etc... into separate tables. Because heaven forbid you just store the HTML itself, or a single serialized array or some such to represent the fields.

Blogger Danby May 26, 2015 7:29 PM  

Just recently I was looking at a registration form for a CMS that generated literally 140 SQL queries just to draw the form.

Or the guy I worked with who re-implemented ftpd in perl.

Blogger Markku May 26, 2015 7:31 PM  

. The developer had come up with an exquisitely-crafted system to decompose every form field, every label, every possible set of values, every form group, etc... into separate tables.

Confession time... I have done PRECISELY that.

Anonymous FriarBob May 26, 2015 7:36 PM  

Heh. I like your rant on #4. All too true.

Though the inverse is also true. The marketing people must ALSO actually listen to the technical people. And interrupting before the second sentence gets halfway done doesn't count.

Blogger Markku May 26, 2015 7:51 PM  

But by amazing coincidence, just this morning I had to touch that system after not having even looked at it for two years. And it did exactly what I intended on the first attempt. My immediate reactioin was, DAMN this system was good.

It wasn't just about generating HTML, it also had to communicate changes to daemons. Different ones, depending on the value. And it could receive changes from those daemons. Also, the allowed set of values could dynamically change in the fields.

Blogger rycamor May 26, 2015 8:03 PM  

Markku, obviously there are times and places to do things like that. Most of the overcomplicated things I find are things that would have a purpose somewhere... sometime. But the developer became so enamored with the idea itself that he didn't bother to ask whether it was practical or even useful for the scenario.

Anonymous DT May 26, 2015 8:12 PM  

Reading and thinking about the comments in this thread I've had a moment of clarity, at least as far as my personal life is concerned.

I love engineering software. But I hate every single thing about the context and culture in which most programmers have to work.

I hate buzzwords.

I hate management.

I hate project management philosophies. All of them except "put intelligent, skilled men together in a room with a problem and let them solve it."

I hate a world that thinks coding is so simple that surely a half dozen highly experienced, high IQ American engineers can be replaced by two dozen code monkeys from a foreign country with 'certificates' from some Chinese, Indian, or Caribbean university that probably doesn't even exist.

And FedGov thinks you're racist if you don't agree.

That same world wants everyone to 'learn coding' and treats everyone with a piece of paper the same. (I cannot express my horror that FizzBuzz is a solid interview challenge because it weeds out 3/4ths of candidates. It is only exceeded by my horror that years after this was discussed openly on the web, it still weeds out 3/4ths of candidates. So not only can 3/4ths of 'programmers' with 4 year degrees NOT program, that can't even Google search interview tips.)

I hate meetings.

I hate the boom/bust cycle driven by all of this.

If I could go back I would have chosen a radically different career path and programmed shareware on the side as a hobby.

Blogger Brad Andrews May 26, 2015 8:16 PM  

Ursus,

At least that environment taught me two very important things: 1) under-promise and over-deliver

Very true. The problem is that many developers are optimists so we think we can develop things much more quickly than we will be able to do so.

Markku,

As for Agile, it is a great thing for what it is: A reaction to the waterfall model. The points are not meant to make sense in isolation, but in contrast to waterfall.

That is part of what I was trying to say in my long-winded reply earlier.

just this morning I had to touch that system after not having even looked at it for two years

I caught flack a few years back because I took too much time developing a system that would figure all the dependencies of the different modules in the system. I am not the fastest, but I make solid stuff.

I was quite vindicated (though not on that review) when someone in that group came back to me about 4 years later to figure out how to adjust that same system. It kept running for years and was straightforward to change for the new need. Perhaps the time spent on solid engineering really was worthwhile, even if management wouldn't admit it!

Blogger Brad Andrews May 26, 2015 8:20 PM  

Good old blog entry on the FizzBuzz problem: http://blog.codinghorror.com/why-cant-programmers-program/

Anonymous DT May 26, 2015 8:21 PM  

Markku, obviously there are times and places to do things like that. Most of the overcomplicated things I find are things that would have a purpose somewhere... sometime. But the developer became so enamored with the idea itself that he didn't bother to ask whether it was practical or even useful for the scenario.

"Look at this awesome framework I wrote for dynamic records! Each record can have a variable set of data. No fixed columns here!"

"The data model for this project is frozen, and every record needs the exact same set of columns. Now instead of select x,y,z with a couple joins we have to join a dozen tables, pivot, and error check for records that do not have a piece of data they should have."

"But what I wrote is so flexible!"

I wanted to kill him.

Anonymous FriarBob May 26, 2015 8:31 PM  

Hmm... Reading some of the anti-Agile comments here, I'm finding myself very much wanting to hear some of y'all's opinions on spiral and/or its "modern" variants...

Blogger rycamor May 26, 2015 8:48 PM  

@DT,

I am convinced that every programmer who stumbles into databases sooner or later conceives of the One True Lookup Table and says to himself "BrillIant! Why isn't everyone doing this?"

Anonymous FriarBob May 26, 2015 8:52 PM  

@rycamor

And then they write it and find out the CMS they designed requires 8 processors and 300+ MB of pre-caching to load up the FIRST page ... after about 5 minutes of processor-churn.

No, I don't have any experience at all with software like that, why do you ask?

Anonymous DT May 26, 2015 8:55 PM  

@rycamor - I think you're right.

Anonymous Jack Amok May 26, 2015 9:37 PM  

I learned very early on that "two weeks" is programmer-speak for "I have no idea how long it will take".

"Two weeks" from me means basically "it should one day, but you are going to interrupt me several times with other requests, one or more of the teams I rely on for data or access to other systems will present an obstacle, needed APIs will turn out to not work or provide output in an entirely inappropriate format, the feature will expose a bizaare bug with the OS, driver or hardware layer that I will have to work around, and it may expose some hidden flaw in our own architecture. Possibly our build system will simply fail for no good reason in the middle of implementation and take time to fix. So I'm going to add a few days."

See, I do have an idea how long it will take, I just don't know exactly what all the time will be spent doing. I do know the time will be stolen away though, so I'm not going to be foolish enough to think a 1 day task will take actually 1 day. The old adage is, take the estimate a programmer gave you, double it, and raise it to the next time increment. I just do the buffering for you, since I assume anyone asking for a feature is foolish enough to believe a programmer when he says "Oh, should only take a couple of hours..."

A 1 day task is a two week task. A 2 hour task is a 4 day task. A one month task is at least two years, if it ever gets done at all...

In all seriousness, the amount of time the average senior developer spends dealing with tangential issues is pathetic, and it's gotten far worse now that Microsoft forgot how to create SDKs and Google and Apple never learned.

Anonymous Jack Amok May 26, 2015 9:48 PM  

Just recently I was looking at a registration form for a CMS that generated literally 140 SQL queries just to draw the form. The developer had come up with an exquisitely-crafted system to decompose every form field, every label...

Huh. I had no idea one of our devs was moonlighting doing CMS systems.

I am convinced that every programmer who stumbles into databases sooner or later conceives of the One True Lookup Table and says to himself "BrillIant! Why isn't everyone doing this?"

I built RDBMS systems and front ends once upon a time. Now I insist on starting off with a no-SQL property-bag style db (Mongo being my personally preference, but it's an open field) and only consider a relational model if and when the no-SQL doesn't cut it, and then only for the specific parts that the no-SQL can't handle. So much complexity gets created trying to decompose things that are supposed to be unified.

Actually, that highlights one of my hard-won lessons in software development: never add a feature or any piece of complexity until you not only know you need it, but you know specifically how it will be used and specifically why something simpler or something that already exists can't fulfill the need.

Blogger Cail Corishev May 26, 2015 10:18 PM  

See, I do have an idea how long it will take, I just don't know exactly what all the time will be spent doing.

Yes. I don't think I've ever had a project that would have taken a full two weeks of sustained coding. That would be a lot of code, and I mostly do smaller one-man projects. But I've also never gotten to code for two straight weeks on a project, because so much other stuff is involved. So "two weeks" often means, "I can probably do what you're describing right now in a few hours, but I don't know how long it'll take you to get me all the info I need, how often you'll interrupt me, how many extra features you'll want to add, how many times you'll change things I've already done, how much time it'll take to show you how to use it.... So I'll say two weeks, and that'll surely be enough, since I can do all the coding on day 14 if it comes to that."

Blogger rycamor May 26, 2015 10:36 PM  

Jack Amok May 26, 2015 9:48 PM
I built RDBMS systems and front ends once upon a time. Now I insist on starting off with a no-SQL property-bag style db (Mongo being my personally preference, but it's an open field) and only consider a relational model if and when the no-SQL doesn't cut it, and then only for the specific parts that the no-SQL can't handle. So much complexity gets created trying to decompose things that are supposed to be unified.


I'm kind of the opposite. I rarely see a need for the NoSQL flavor of the day, because I have actually mastered the concept of relational, and understand that it can be used in many, many more ways than most developers conceive of. Of course, I use a DBMS that allows me to out-NoSQL the NoSQL databases when I want to.

It really depends on the applicability of the logic involved. If I were creating a user-customizable form system that allowed one to create complex business rules around them, I might decompose everything in the way mentioned above (although I would also have cached version of the form so every request wouldn't re-query all tables).

But most likely I would take the approach that any one form is a complex value, and store it in a JSON or XML field (meaning values within it can be accessed via query, but it exists as one complete unit). This would result in only ONE query to render the form, but I could still express logic regarding the form. This is completely in accord with the relational model, as there is nothing preventing any one value in a row/column from being complex in nature, as long as it is encapsulated at the row/column level.

Blogger MidKnight (#138) May 26, 2015 10:53 PM  

God.

That looks like yet another list of cargo-cult bullet points, that started out as an actual "good idea" for certain types of development work, and mutated and was misapplied from there.

Back in the 90's when they were trying to implement process improvement/TQL whatever in the Navy, my comment after looking over the class was "the good stuff here - including "listen to the guys doing the job for suggestions and hints of hangups" - are basically case-specific iterations of classically known leadership principles.

The actual iterative improvement cycle actually was sort of novel - at least to factory and design improvement by gradual refinements. Make a change, stabilize, see how that holds out. Tweak again to bring things under a higher degree of precision. Yay.

Hell - anyone who'd run a compression-cycle distiller knows THAT. Get it in the overall zone, and then make a tweak, and WAIT. Make another tweak. WAIT.

Dilbert in that era wonderfully spoofed the cargo-cult lip-service to management fads, executed devoid of actual leadership or understanding of the reasoning or meaning behind it.

Blogger Harsh May 26, 2015 11:27 PM  

Why do you find it hard to tell them "I don't know"? I mean, you do realize that within two weeks they're going to find out that you don't know what you're talking about, right?

It's amazing how many people in the business world think you should never say that you don't know something. Like everyone is supposed to have an answer for everything always. I probably respond to questions with "I don't know" more than anything else, but I usually follow it up with "but I'll find out".

Blogger Brad Andrews May 26, 2015 11:39 PM  

Rycamor,

I would lay good odds nothing will ever be at your home page. I think 9 years is enough time to get something there!

I would love to find out more about the trends you are seeing. Send me an email to the account linked here if you want to do that.

Anonymous Jack Amok May 27, 2015 12:17 AM  

I'm kind of the opposite. I rarely see a need for the NoSQL flavor of the day, because I have actually mastered the concept of relational,

Well, I did as well, over two decades ago. But I think RDBMSes are one of a category of things I call Complexity Pits. They suck in the inexperienced coder with a false sense of "correctness." Meaning it's easy for them to think - as the guy decomposing everything in the story above - that they need to create a bazillion tables and a bazillion-squared indexes and that there can never be duplicated data...

The classic facepalm is of course when a noob creates a PO/Inventory system with the price referenced from the products table, and then someone finds the bugs reconciling old invoices after a product when on sale... Sure, it's a trivial example, but it's very much the sort of mistake encouraged by the relational mantra. Where as with a property-bag, of course the price paid is stored with the order. Everything is stored with the order...

RDBMS has it's place, but I believe that if I stress with the junior programmers that a non-relational DB is the preferred solution in general, they're less likely to sink into the pit. Complexity is the enemy, and it multiplies fast if you don't actively oppose it.

It really depends on the applicability of the logic involved. If I were creating a user-customizable form system...

User-customizable form system? By any chance did you ever use an ancient program called AccessSQL? (no relation to the Microsoft's Office DB, this one came from an outfit called Software Products International).

Anonymous rho May 27, 2015 1:09 AM  

I'm kind of the opposite. I rarely see a need for the NoSQL flavor of the day, because I have actually mastered the concept of relational, and understand that it can be used in many, many more ways than most developers conceive of. Of course, I use a DBMS that allows me to out-NoSQL the NoSQL databases when I want to.

If you have a project where you need to store and access formalized data, an RDBMS is everything. If the data were not important, you can use anything you like, but if the data are important, it's nice to have something you can have faith in. It's also nice when you're suddenly asked to do something unexpected and you know you have a real, reliable and well-supported platform to build on.

Keeping presentation information in an RDBMS is kinda daft, but I could see the need to do so on occasion. 140 SQL queries for a form is super dumb, though. If it's not wrapped in a transaction, you're going to have consistency issues. If the data doesn't change enough that consistency won't be a problem, you're using the database incorrectly.

My earlier comment on Agile got eaten, but the gist was that Agile is probably good because practically any structured and formalized process will probably work well enough to be considered good. As long as it doesn't require gladiatorial combat or ritualized sodomy, it's likely Good Enough(TM).

The main purpose behind these faddish managerial techniques is less to organize the team and more about bashing the project into some form that can be consumed by a development team. Linus ran the Linux kernel development for 900 years using only clay tablets and contributors submitted patches via semaphore. This worked because Linus is very smart, the kernel core is organizationally very flat, and the projects limits are by definition finite.

(On the LKML, sodomy isn't ritualized, but it was definitely recreational.)

Anonymous Jack Amok May 27, 2015 1:27 AM  

Hell - anyone who'd run a compression-cycle distiller knows THAT. Get it in the overall zone, and then make a tweak, and WAIT. Make another tweak. WAIT.

Anyone who's done any of several actual mechanical things would know a lot of things that are great surprises and revelations to many people in the software industry. The problem with software is you don't see it work. Maybe the developer sees his own code, sort of, stepping through it in the debugger. But the guy badgering him for new features or asking for some bug to be fixed really fast for a complaining customer, they don't see, feel or hear squat. All they see is the programmer waving his finders at a keyboard for a while and then the program does something...

It is akin to magic to them, and they literally have no clue how long anything takes or why it takes that long. I'm sure it's a frustrating existence for them, but I do wish they'd learn to live with it and stop pestering those of us who do know.

Regarding the FizzBuzz post...

I cannot relate. While I have interviewed candidates who could not solve a problem like FizzBuzz, they were far from the majority. I don't know where these people are digging up their candidates, but they need to find a better screening process. The majority of developers in the world are not great, but that's just sad.

OpenID joshtheaspie May 27, 2015 2:15 AM  

Regarding your thoughts on #4... Maybe. I'd propose that if anything in this edge (permaculture term) would improve things, it would be open, and most importantly \honest\ two-way communication. One of the reasons that engineers tend not to listen to marketers, is that marketers tend not to know about tech, while pretending they do, and ignoring what the engineers are telling them.

Simultaneously, the techs get told to do just as much work (if not more) with a team that suddenly consists of half as many people by the business people.

So the message most techs get is that business people and marketers know how to shmooze, and make up new buzz words (like agile), and are otherwise dishonest idiots that over-promise on things they don't understand. But hey, it can't be their fault. They instituted processes.

Techs want to be able to regularly check results with people that will actually use the software/hardware, to ensure that it is meeting un-written expectations. What they often get is occasional directives from a guy who doesn't actually use the software, on how it should operate... which is often out of line with how it actually needs to. Something you alluded to above.

Blogger JP May 27, 2015 5:13 AM  

Our best, most successful, on-time, on-budget projects were ALWAYS the ones where we were left alone to produce the end result without lots of management overhead.

On the other end of the spectrum, I worked on an AGILE project two years ago along with three testers, four project managers, a client rep and some bitch named Susan. It was basically a directory of beauty salons with a map, location-aware search and a back-end where people could list their specials of the day. It took 7 months to complete and half the people involved quit because we had to work 10 to 16 hours per day, 7 days per week to keep all the "stakeholders" happy and the amount of finger pointing was unbearable. The best thing about it was that the client was allowed to make architectural decisions.

Using our own "primitive" model, we could have produced the same solution within 2 to 4 weeks in a team of 2 developers and a manager who does nothing more than listen to what the client wants, discuss it with us and then leaves us alone to do it.

Anonymous Jack Amok May 27, 2015 5:24 AM  

Using our own "primitive" model, we could have produced the same solution within 2 to 4 weeks in a team of 2 developers and a manager who does nothing more than listen to what the client wants, discuss it with us and then leaves us alone to do it.

Look, I know it seems like it ought to be that simple, but if you tried to do it like that, you might solve the customers problem, but you'd be doing nothing to solve the problem of finding remunerative employment for project managers, client reps, and bitches named Susan. These people have no useful skills, you can't expect them to find work anywhere else. That's why people always say that engineers don't make good managers. Engineers don't find enough patronage jobs for Ivy League midwits and the CEO's former mistresses who have incriminating photos.

OpenID newscaper May 27, 2015 8:22 AM  

We're almost 2 years into Scrum insanity, those of us still left who know anything about to give up any lingering hope that mgmt will realize it's a terrible fit for certain projects, and alienates your sharpest people.
Most dishonest thing about whole 'transformation' was the sales pitch never veered from Scrum/Agile vs waterfall, even though I explained to the Worlds Greatest Consultant we were never the proverbial narrow spec coders of waterfall. We'd relied on some very sharp people back in the day to create the core systems, now legacy. We needed to add a little of the right structure to ad hoc, coming at it from other direction. Instead, we ended up with a bizarre, even more micromanagement prone variation on Scrum.
We were small enough key devs had a foot firmly in the analysis side of things as well, but now we're supposed to happily retreat inside our new little box inside which we are 'empowered', pulling tickets from the backlog by someone declared Product Owner by fiat, but actually knows LESS about the business than most of us, because the magic process defines one. Instead of PO role bringing biz and tech together, they've created a distinct job and injected a non value adding middle man. So Know have the pleasure of explaining everything I need to do to PO so she can write it up and to higher mgmt it looks like she's doing a bang up job of directing me.

Worse, on project that is vital to the company, re-imagining and re-architecting a horribly decrepit legacy system, where I'm the guy with the vision and ability to basically invent a new approach, in this stupid scrum model that person cannot even exist.

Blogger Lew Rand May 27, 2015 11:38 AM  

Engineers don't find enough patronage jobs for Ivy League midwits and the CEO's former mistresses who have incriminating photos

And once those people get removed, you start looking at the CEOs and CTOs and other Cs and ask why are we paying a million dollars to someone to run a company of 10 people...

Blogger Markku May 27, 2015 12:24 PM  

I think middle management exists, not because it does any real good, but as a convenience for the owner to avoid having to deal with the serfs. It functions as a layer of social isolation.

Blogger JaimeInTexas May 27, 2015 12:53 PM  

8O Good God! No-SQL for a PO system?

RDMS does not stop anyone from de-normalization. It is a proper technique, such as when seeding a product price at time of a purchase. Depending on the business process, editable until a certain status is achieved, after that, changes may be allowed with a review process and auditing trail.

Microsoft SQL has indexing (column stores) that remove the need for explicit de-normialization.

Blogger Markku May 27, 2015 12:57 PM  

Jack didn't say it did. Just that it encourages wrong thinking about the system. For example, if you think about it, an invoice shouldn't actually point to the present state of the product, because what was actually shipped was its state at a fixed point in history. At most, you might want to attach it to what product you would replace it with, should the customer claim warranty at some point in the future. And that probably should point to the table called "products".

But this is obvious only from experience. Usually by making that mistake. Been there...

Blogger JaimeInTexas May 27, 2015 1:20 PM  

should have stated: Microsoft SQL has indexing (column stores) that remove the need for explicit de-normialization in some cases.

Blogger rycamor May 27, 2015 1:28 PM  

Yes, neither NoSQL nor classic 3rd (4th, 5th Boyce-Codd) normal form will present you from making serious mistakes in business logic.

AFAIR, C. J. Date (one of the big names in relational theory) used to start lectures by saying something like "It is important to think as clearly as possible about how we want to use our data." At one of those moments a (female) attendee raised her had and said "why is that?". He almost considered quitting at that point.

Blogger JaimeInTexas May 27, 2015 1:31 PM  

'At one of those moments a (female) attendee raised her had and said "why is that?"'

LOL,

Blogger rycamor May 27, 2015 1:56 PM  

Brad Andrews May 26, 2015 11:39 PM
Rycamor,

I would lay good odds nothing will ever be at your home page. I think 9 years is enough time to get something there!


Heh... I had totally forgotten about that. I recently started using my Google login here because of the Captchas.

Blogger rycamor May 27, 2015 2:04 PM  

Brad Andrews May 26, 2015 11:39 PM

I would love to find out more about the trends you are seeing. Send me an email to the account linked here if you want to do that.


I clicked on your profile but saw no email. You can contact me at my handle at Gmail.

Anonymous Jack Amok May 27, 2015 9:55 PM  

Jack didn't say it did. Just that it encourages wrong thinking about the system.

Exactly. Why does a particular record need to be stored in an orderly row/column format? Most real-world examples of things aren't quite so orderly and frankly are better represented by a generic dictionary object than the formal grid that RDMBSes encourage you to think of.

But some things are better represented in that formal grid. If you can articulate why a relational DB is a better store than a property bag/document system for some specific use, then by all means use the RDBMS. But you shouldn't default to it. Default to the straightforward thing, and worry about cross-indexing and strong typing and the rest only if and when you need it.

Even a PO system may very well be better done as a no-SQL. Not always, but sometimes. Not many on-line games that let you buy power-ups, skin packs, deco items and the like - i.e. that have a point of sale system - use SQL back-ends, because building the POS system isn't what they're about - they're trying to make a game and will use whatever works cheaply and reliably. If you need cross-indexed reports, just use a MapReduce operation later.

Blogger JaimeInTexas May 27, 2015 10:50 PM  

RDMS tables and a stored procedure that returns multiple, hetergeneous, recordsets accomplish the same thing as a property bag or document. The client application still has to know how to use the data.

Anonymous Jack Amok May 28, 2015 4:06 AM  

RDMS tables and a stored procedure that returns multiple, hetergeneous, recordsets accomplish the same thing as a property bag or document.

Sure, in a more complicated manner. That's the whole point. Why build a Rube Goldberg device to make toast when you can just push the damn level on the toaster? Just because a tool can be made to do a job doesn't mean it should be used for it.

If all you need is a property bag, then use a property bag, don't create some relational rats-nest behind an interface that mimics a property bag. Save the added complexity of a relational store for places where it does some good.

Blogger JaimeInTexas May 28, 2015 10:31 AM  

Amok. I am really not totally disagreeing with you. I am definitely in the "do not reinvent the wheel" camp.

There is, however, knowledge that once acquired can be replicated at will, having built a structure to accomplish the task and having total control of the internals, well, there is much advantage gained. I do not think that it is more complicated. It is all in the implementation, I suppose.

I do think that a no-sql PO system, for example, would have to rely way too much on client side logic. Am I wrong?

Anonymous Jack Amok May 28, 2015 12:39 PM  

I do not think that it is more complicated. It is all in the implementation, I suppose.

More in the interests of making my position clear than in simply arguing with a fellow minion...

Many developers make that same claim. Some of them are correct. It's a common conceit among people who do what we do for a living to think they are smarter than they really are and find ways to favor "clever" solutions that demonstrate their knowledge. That same "knowledge that once acquired can be replicated at will", regardless of whether the knowledge is appropriate to the solution it is replicated into. If the internal controls of a relational system are useful in the new solution you have replicated it to, then great. But are they? That's the question I want someone to ask before throwing an RDBMS at a solution.

Because if those internal controls are not perfectly appropriate to the problem you are trying to solve, then you (or more likely, the poor client-side engineer) will have to implement additional client-side logic to work around the obstacles the DB creates with inappropriate server-side logic. Furthermore, the reason for that additional (likely very ugly looking) client-side logic will be totally opaque to someone looking at it six months from now when they need to fix a bug.

I do think that a no-sql PO system, for example, would have to rely way too much on client side logic. Am I wrong?

Logic is logic, regardless of where it goes. The question is where the logic is most cleanly implemented, where it is most adaptable, and where it is least prone to errors and easiest to debug. Often that is behind an Insert() function, but there's no reason the Insert() function has to be at the db store level. It can easily be in the layer that wraps the db. Most SQL-based solutions I have seen used an adapter layer anyway, and putting the logic that separates out, say, line-items, for their own DB, could just as easily be in the adapter, where the programmer can see it and walk through it.

Sure, the DB can do that for you without having to write additional code for the insert, but only if you've already done the work somewhere else to define the structure. Whether you are saving or creating work with one solution or the other is a question that has to be answered by evaluating the specific problem.

Another consideration is that using a SQL solution requires your staff to have SQL expertise. They already need expertise in whatever language the front end and non-storage server pieces use, but by using SQL, you've added the need for another knowledge base. As a sql expert, you may like that, but as someone responsible for the staffing, schedule and cost of a project, I don't. I prefer reducing the amount of specialized knowledge necessary, applying it only to where it has a clear advantage to the project.

Anonymous rho May 29, 2015 3:49 AM  

Old thread is old, but:

Sure, the DB can do that for you without having to write additional code for the insert, but only if you've already done the work somewhere else to define the structure. Whether you are saving or creating work with one solution or the other is a question that has to be answered by evaluating the specific problem.

They're called stored procedures. Call me when you need meaningful statistics from your "property bag".

Anonymous Jack Amok May 29, 2015 11:05 AM  

They're called stored procedures. Call me when you need meaningful statistics from your "property bag".

That's what mapReduce is for.

And these "stored procedures" that you think I am unfamiliar with (insert laughter here)... tell me, do they write themselves? Debug themselves? Explain themselves to the new hire who comes on board ten months from now? Or are they - like any other piece of coded logic - something with costs and benefits that must be weighed against each other?

Post a Comment

Rules of the blog
Please do not comment as "Anonymous". Comments by "Anonymous" will be spammed.

<< Home

Newer Posts Older Posts