Programmers Share The Biggest F*** Ups They've Seen That Resulted From A Tiny Error

"Please Try Again"

A co-worker made a change to our credit card processing code on a Friday evening, didn't test it, pushed it to prod, and then went home.

He made a typo in his code that caused a runtime error on the back end. Everyone that tried to make a purchase received a generic "Couldn't process your card. Please try again." But it did process their card... it was failing at some step after. People would retry several times and their credit cards were being charged every time.

Super cluster fck on Monday morning.

- fade_00

Leave

There was a leave application at a company I was working for, it was a nice 30k employee strength software company.

The leave application had a workflow built in. If an employee's leave wasn't approved by his manager for 3 days it would go to the next level manager and stay there for 5 days and so on..

To test some aspect of this workflow feature, the approval waiting time was reduced from x days to 3 mins. Then without reverting this test change the code was pushed to production.

The next day, the company's CEO had around 500 leave applications in his mailbox.

- dominus_selenos

The Wildcard

One of our engineers, when writing some custom scripting for a customer, made the decision to use what's called a wildcard character in the delete statements. For those who don't know, what these characters do is let you do comparisons that do not need to be an exact match, such as using "*ing"

The system would then match any value that ends in "ing". Besides the already WTF of using a wildcard when trying to delete they also did not validate that the data being supplied was present. So basically what ended up happening is the customer processed an order where this data was empty, causing the wildcard to be all by it's lonesome and matching with literally anything.

The end result was about 6 months worth of orders being deleted.

- Kirgio

Employee #1

I was on-call for a retail company when, early one morning, we had a problem where inventory couldn't be taken in any of our stores. Tracked the problem down to 25-year-old code written for the hand scanners that wasn't in source control.

Turns out, the startup code did a check to validate the database connection; the way it did this is by looking up employee #1.

She'd retired a week earlier and removed from the database the previous evening, causing everything to crash like dominoes. The company was going to replace the scanners at some point, and we had no way to build a Java version that old, so they re-added her info and added code to exclude her as a recognized employee for discounts/etc.

- cjcmd

Praying

I used to work for a company that got people (mainly Americans) online and sent out tons of CDs as ways to sell their service. There were a bunch of scripted jobs that you could use to restart servers, as our naming conventions were pretty good (so if you wanted to restart all the webproxy servers, you could run `./script webproxy*` and have all those servers restarted. Good for code upgrades or patches, or whatnot. Script worked off of a big inventory list and could restart multiple "classes" of servers in case you were doing work on an entire subsystem.

Someone ran it like so:

script webproxy *

Notice the space. The script began restarting every host everywhere. One of the VP's was in the war room trying to fix things, was offered a chair (as he was kneeling in front of a keyboard) to which he responded "No, I'm praying" and kept on working.

- faisent

Do The Robot

During work experience, me and my friend went to the schools IT department and had been tasked with building and programming robots as well as making a presentation on how to recreate our results. They were remote controlled except they all shared the same radio channel so due to a single line of code 1 controller controlled 8 robots.

So we turned off the lights and made them dance with flashing disco lights on them.

- gaVtehchAV436

Identity Thief's Playground

I do a lot of internet screen-scraping, and was getting healthcare provider information from a large insurance site. I got to the listings of physicians and found that the parameters to see a physician's details were made up of a giant database dump of information. It had the doctor's name, office address, phone, SSN, birthday, home address ... it went on and and into tiny details. One wouldn't see it on the site, but click on the source, and you it was an identity thief's playground.

I called the insurance site immediately, but no one could get me to a web-admin or IT, so I looked up the CEO's number, and called there. I talked to an executive assistant. I told her if it wasn't fixed in a day, I would need to start calling the affected doctors.

The site was down about a day and a half later, and after a few days returned without that egregious error.

- SurlyJason

Fish Tank

I'm learning python and micropython right now.

- benign_saidbenign_said

Molten Aluminum

At long-defunct car parts manufacturer. A big robot was taking a crucible of molten aluminium and pouring it into molds. Suddenly there was this loud whine of rapidly moving servos, and it threw the crucible of, I repeat, molten aluminium through the roof.

Later I asked one of the engineers what had happened. "We lost the tacho" he said, laconically.

In more detail, they lost the signal which told the controller how far the arm was moving - loose wire I guess. So the controller decided to feed in more power to make it move. Lots more power.

Apparently you could tell the experienced people because they were the ones under the benches as soon as they heard the whine.

- thx1138a