T-SQL Tuesday—Bad Bets

The year was 94, in my trunk is raw, and in the rear view mirror was the mother $%^in’ law. Got two choices y’all pull over the car or bounce on the devil, put the pedal to the floor” – Jay Z 99 Problems

Ok, so the situation I’m going to be talking about isn’t as extreme as Jay’s of being chased by the cops, with a large load of uncut cocaine in my trunk. The year was actually 2010, things had been going well for me in my job, I had just completed a major SQL consolidation project, built out DR and HA where there was none before, and was speaking on a regular basis at SQL Community events. A strange thing happened though—I got turned down to go to PASS Summit (in retrospect, I should have just paid out of pocket,) and I wasn’t happy about. All of the resources in the firm were being sucked into the forthcoming SAP project. So, back when I thought staying with a company for a long time was a very good thing (here’s a major career hint—focus on your best interest first, because the company will focus on its best interest ahead of yours), I decided to apply for the role of Infrastructure Lead for the SAP project. I’ve blogged about this in great detail here, so if you want to read the gory details you can, but I’ll summarize below.

The project was a nightmare—my manager was quite possibly mental, the CIO was an idiot who had no idea how to set a technical direction, and as an Infrastructure team we were terribly under-resourced. The only cool part, was due to the poor planning of the CIO and my manager, the servers ended up in Switzerland, so I got to spend about a month there, and work with some really good dedicated folks there. I was working 60-70 hours a week, gaining weight and was just generally unhappy with life. I had a couple of speaking engagements that quarter the biggest of which was SQL Rally in Orlando—I caught up with friends (thank you Karen, Kevin, and Jen) and started talking about career prospects. I realized how what I was doing was useless, and wasn’t going to move my career forward in any meaningful sense. So I started applying for jobs, and I ended up in a great position.

I’ve since left that role, and am consulting for a great company now, but almost every day I look back fondly at the all the experiences I got to have in that job (the day after I met my boss, he offered to pay my travel to PASS Summit 2011) and how much it improved my technical skills and gave me tons of confidence to face any system.

The moral of this story, is to think about your life ahead of your firms. The job market is great for data pros—if you are unhappy, leave. Seriously—I had 5 recruiters contact me today.

T-SQL Tuesday #44 Second Chances

Bradley Ball (b|t) who also happens to be my moderator for the upcoming 24 Hours of PASS is the leader for this month’s T-SQL Tuesday, which is all about second chances. When I think about things I’ve screwed up in my career, and there are many, I always fall back to one, and it doesn’t even involve SQL Server, but another RDBMS and it gives a couple of lessons on how to be good DBAs.

It was late on a Friday afternoon, 1522 to be exact (here is lesson #1—never do anything involving production on a Friday afternoon unless you have to), and I got a call from a user, asking me to refresh the QA database with production data. This system was an environmental monitoring system, that monitored the atmosphere and surfaces for the biopharmaceutical manufacturing plant that I worked in, it wasn’t exactly mission critical, but it was still pretty important. Since the user was a friend of mine I jumped right on it. In the database I was working on, as opposed to what I’d probably do in SQL Server (which would be to restore a backup) there is a very easy to use import/export feature, that allows for easy logical restoration of a specific objects/schemas, etc. So this was my standard methodology for doing a refresh with prod data, but I had yet to script it (lesson #2—be lazy, if you have to do something more than once, script and automate it).

Anyway, I go ahead and take my export of production, and start to import back into the QA environment. Typically, my process would be to log into QA, drop the user that owned the objects, and then run the import. For whatever reason (and in this case it was probably a good thing), I didn’t do that. I started my import and noticed I was getting some errors—once again, not something I’d ordinarily do, but I cancelled the job and reran it with error suppression on (lesson #3—always read the errors, and never turn error suppression on). The job completed without error, I emailed the user back telling him that it was complete, and I went on with my Friday afternoon. About five minutes later, I got a phone call from the same user:






There’s this classic moment that happens in IT (and probably happens in each of these blog posts), that I like to call “the bead of sweat moment”, it’s that moment you realize you #$%ed up badly, but no one else has quite realized that you are responsible yet. I asked the user to get everyone out of the system. In ordinary companies an error and outage like this would not necessarily be a big deal, but this a control system in a pharmaceutical plant, so it was heavily regulated. What had happened was that I (accidentally) did an import from prod onto itself, those errors I suppressed were duplicate record errors.

So now, I had a production database filled with duplicate data. So I go tell me boss—I screwed up, and we’re going to be down for about 10-15 minutes. Fortunately, I knew from the log of the job, the exact time the records were inserted. Also, the database was in the equivalent of full recovery mode, so I was able to do a point in time restore to the second before I started the import job. This leads us to lesson #4—always have a backup, and even better if it’s near to the system (in this case it was on local disk, before it was zipped off to tape).

The lesson I learned were several, but the biggest is that if you have good backups (and regularly test your restoration process) you are protected from a lot of evils.

T-SQL Tuesday—A Day in the Life


Last week Erin Stellato (b|t) posted a great suggestion for T-SQL Tuesday topic. A day in the life—basically chronicling what we do on a given day and writing about it in or more detail. My job title is Principal Architect-SQL Server, but in reality I cover a lot more than that. There are only two members of my team who really work with Windows, so we end bouncing a lot of ideas off of each other. Additionally, I’ve also worked a lot on that other database platform, which lets me interact with the Oracle folks on my team and even get in and do some testing from time to time. All that being said, my job is still kind of vague, we work to write a lot of documents, set best practices, and engage the teams we service, with varying degrees of success. I don’t carry a pager anymore, but the problems I attempt to solve tend to be a lot bigger then fixing a single database problem.

So here is my Thursday July 17, in all of its glorious detail.

5:30 AM—I’m up early, as usual. After reading some email on my phone, I make a cappuccino to get the blood going. I go out for a short (16 mi/25 km) bike ride, legs feel really good, and my power output is pretty good. Get back and take quick shower.

7:30 AM—Usually, I work from my office, which is a lovely 5 min (5 km/3 mi) commute from my house. But this morning I have a meeting at our corporate headquarters, which is in Center City Philadelphia. This means two things 1) I need to take a train to the office 2) I need to iron a shirt, because their dress code there is more stringent than my office. So I iron a dress shirt, pack my laptop bag and drive to the train station.

8:30 AM—I’m taking Amtrak into the city—the Amtrak train has WiFi, so I can catch up on and send some emails before I get to the office. I also, start working on my status report.

9:20 AM—I arrive at the our Corporate HQ, one of my teammates is working from home today, so I go up to the 40th floor and hijack his desk. I complete my status reports and have an IM exchange with one of my storage colleagues about some testing we are performing with a new SAN vendor. This will take up most of my afternoon.

10:00 AM—I have another conference call with a storage engineer, about the potential of creating a SMB share for a multi-site AlwaysOn cluster I am helping to design. There is some confusion about the concepts of a multi-site cluster, so I break out my handy Visio diagrams, and we reach mutual understanding.

10:30 AM—Meeting with a Systems Architect from a Major Software vendor and one of our lead database architects about our goals and strategy in coming years as they apply to databases. Most of this meeting was NDA, but it was quite entertaining, and we got a lot off of our chests. I was hoping this meeting would end a bit early, so I could catch a 12:15 train back to the suburbs, and be at my storage test at 1300, but no such luck. I check the Amtrak schedule on my phone and see there is a 12:45 train that I can make, that will only leave me a few minutes late for my meeting.

Note: Normally if I have to go to my office in the city (which is like 1-2x/month, typically), I spend the whole day there. Today, I had to get back to my normal office for some meetings with a SAN vendor, but I really wanted to attend that architecture meeting in person.

12:00 PM I contact one of the Oracle guys on my team, and ask if he can go down to the SAN meeting, as one of my tasks was going to be to install Oracle on the test servers. He had the time, so I didn’t have to sweat being a few minutes late.

12:05 PM I grab a quick sandwich from the food court in our building. Normally, I try to have a really light lunch as I’m trying to lose some weight, but since I was probably doing 2 bike rides today, I was happy to indulge in a pulled pork, provolone and collard green sandwich from Percy Street BBQ. I sit down and have lunch, and there a two college age guys next to me. One of them spent the whole time talking and trashing his girlfriend. Classy. My favorite quote was “she’s good enough for right now”. Dude—you’re a jerk, and you aren’t that hot either. Rant over.

12:45 PM Catch Amtrak back to the burbs, with 5 min to spare (why is it you are always running in a train station)

1:15 PM Get back to car, proceed to drive back to West Chester office.

1:30 PM Walk into storage meeting right on time—nothing is configured enough to begin testing, but we talk a little more and go through some features of the hardware.

2:00 PM Hardware vendor goes off into unrelated sales pitch of another product. I happened to sit through this sales pitch last month and thought the product was a steaming pile. I’m not happy at this point and proceed to vent on twitter. Followed by catching up on some SQL things I had been working on in the morning.

2:45 PM I sneak out of the conference room, and go upstairs to catch up on news with my Windows colleague.

3:00 PM Go back to conference room, and figure out the the vendor needs our test hardware for a demo they are doing tomorrow, so we can’t really do anything yet. Continue to get frustrated, but also work with the vendor to have a deeper understanding of their product.

4:50 PM After a long afternoon, I take off—I was hoping to get more accomplished with this hardware test, so we are a couple of days behind on that.

5:05 PM Get home, change directly from dress clothes to cycling clothes, I eat a piece of fruit and fill up water bottles.

6:00 PM Go to training race that I do on Thursdays. It’s been a big step in my fitness to be fast enough to complete this race, and I’m really starting to enjoy it. Not enough to get the racing bug again, but enough to feed my competitive needs.

7:25 PM Get home from ride. I’m generally really into food, but after a big day and a pretty hard ride, I don’t want to eat much. So I have a bit of pasta, watch the tour (that’s the Tour de France, for all of you non-cyclists), and catch up on email.

This was a pretty long day, and slightly atypical, though not that far from normal.

%d bloggers like this: