Certified What?

Well folks. I actually did it, I finally achieved the Microsoft Master's designation after passing the Cert Lab. Ryan Conrad, MCM Program Manager sent me the good word today.

Now what do I do with this new found acronym? Ask my boss for a raise? Well I am self employed, so that won't work. Shoot for the Microsoft Certified Architect certification? Uh, that would be a big NO at this point.

Honestly, I feel like running ten miles just to stop the adrenaline flow through my body. What a rush! And after that I feel like sitting in front of the TV for a day with just the clicker in hand. Once I am acclimated back to my normal environment, maybe it will sink in then.

To all of the guys from my rotation who are still working on achieving the MCM, please don't give up! Keep at it! You can do it, I know you can!

I will never do that again...

Have you ever woken up after a long night of drinking, mouthing those words to the porcelain God?

Well I am typically the kind of guy that does a bad thing one time, like sticking a butter knife in a wall socket - been there, done that - once!

Yesterday, I did a bad thing twice - I retook the MCM Active Directory Lab again, although this time from the comfort of my home. The second time was a little less intimidating, but not painless. I had nightmare flash backs of that first dreadful day in Redmond. The nine hours dragged on while the familiar sound of audible cursing filled the room when my screen didn't refresh fast enough and when a normally simple configuration task turned into a programmed and devious diversion of "not so fast, you just thought that was going to be easy." I have to hand it to the MCM lab designers, if this were the Middle Ages and the Inquisition was recruiting, these guys would have job security.

Well, that was yesterday, and today I am feeling better...waiting for my results. Looking back at it, maybe it wasn't as bad as I am imagining. On second thought, it was, but I am not one to leave business unfinished and I paid out of pocket for the electric shock therapy, so NOT retaking the lab was not an option. I am as my wife says, the most stubborn person on the face of the earth - persistent is probably a more euphemistic word.

Crossing my fingers! Will let you know as soon as I get the word.

Nursing a Sick DC Back to Health

In Part 1 on DC health, I described the symptoms of a DC that has fallen ill, preventing it from replicating with its partner DCs.

So how do you nurse a sick DC back to health? Triage your actions depending on the symptoms.

Symptom: Schema Version Mismatch

As I stated in the previous article, this may not necessarily be a symptom of illness but the normal process of DCs making sure that they have the latest copy of the AD building code (schema) before replicating from a partner DC. In other words this is a normal behavior that occurs when the schema is updated and it takes time to replicate the schema changes throughout the AD Forest.  The only action required is to wait for a normal replication cycle to take place or if you are not the patient sort, force AD replication to occur.

To find out the Schema version of the Forest, use either ADSIEDIT.MSC or DSQUERY from the Schema Master DC.

1. Using "ADSIEdit.msc " or/and "LDP.exe" tools:

 Navigate to: "CN=Schema,CN=Configuration,DC=domain,DC=local" and review the current "objectVersion" attribute.

 2. Using "DSQuery" command line:

The following is a list of the schema versions broken down by operating system release but all DCs in the same forest, regardless of OS, should be at the same schema level.

13 -> Windows 2000 Server

30 -> Windows Server 2003 RTM, Windows 2003 SP1, Windows 2003 SP2

31 -> Windows Server 2003 R2

44 -> Windows Server 2008 RTM

47 -> Windows Server 2008 R2

Symptom: Lingering Objects

You have basically two options when a DC comes down with a case of lingering objects: either remove the lingering objects or allow the lingering objects to “re-replicate” to the other domain controllers. Which action you take depends on a registry value “Strict Replication Consistency” that is set on each DC.

Think of “Strict Replication Consistency” as a country club restaurant where you would like to take your family for brunch on Sunday. When you call the club restaurant to make a reservation, the Maître D will ask for your name. If the club has strict rules for who is allowed to brunch there and you are not already on the club membership list, you will be re-directed to the nearest Waffle House. If the club has loose rules, and allows anyone to eat at the club restaurant, then your name will be added to the reservation list and you will be welcomed. Similarly a DC either takes a strict or loose stance against allowing lingering objects (non-club members) to replicate to it.

Windows 2000 DCs up until SP4 were always loose and crazy guys, never taking a snob approach when detecting a lingering object. They had a big heart and let anyone in the doors.

Windows 2003 DCs and later are the snobs of the family where strict replication consistency is enabled by default and  if a lingering object is detected, they will refuse to replicate it.

Now, being a snob can be beneficial at times. In the case of lingering objects you are preventing objects that have been purposely deleted from coming back. Wouldn’t you feel a little irked if you kicked a member out of your country club because they couldn’t play nice but then you have to sit next to the guy at Sunday brunch because of loose club restaurant rules?

Imagine now that your predecessor was a bad boy admin and was fired and escorted from the building and their AD user object deleted. You probably don’t want to reintroduce the bad admin user account back into AD where they could logon and wreak havoc. Strict replication rules do have their place under the sun.

Enforcing strict replication consistency is the default and the only time you would want to disable it is to purposely reintroduce lingering objects back in the directory like reinstating a club member you previously booted out. In most circumstances, lingering objects should be removed as they were intentionally deleted for a reason.

If you go against your better judgment and want to turn of Strict Replication Consistency, you can do so with REPADMIN.EXE.

As you can see in the following graphic, you can enable or disable strict consistency on all DCs at the same time.

Before you decide on your next action step, you should first determine why lingering objects are present to begin with. The two primary reasons lingering objects show up are because a DC was restored using an unsupported method (i.e. a virtual machine snapshot causing USN rollback) or the DC failed to replicate for longer than Tombstone Lifetime (TSL). The latter is not so bad, as the DC like Rip Van Winkle is stuck in a time warp that is going to require some “reeducating” while the former is a much more serious problem that involves “zapping” the DC with a “Men in Black” neuralizer.

If it is a TSL issue and you need to erase from the Van Winkle DC, knowledge of objects long since deceased, you can remove the lingering objects with either REPADMIN.EXE or REPLDIAG.EXE.        

REPLDIAG.EXE is the preferred tool as it can clean all Naming Contexts (NCs) from all DCs with one command REPLDIAG /RemoveLingeringObjects. You can download REPLDIAG.EXE from Codeplex at http://activedirectoryutils.codeplex.com/releases/view/18287

If using Repadmin.exe:

1.       First find the GUID of the DC with the known good replica with REPADMIN /SHOWREPL DCname.

a.       So if DC1 has the good replica:

2.       Then run Repadmin /removelingeringobjects  DcNameThatMayHaveLingeringObjects GUIDofDCthatHasAKnownGoodReplica  DomainNamingContext

3.       Check the Directory Service log of the DC that may have the lingering objects for events 1937 and 1939 to see what was removed.

4.       Repeat for each NC (Schema, Configuration, Domain, etc.)

After removing the lingering objects, you will need to restart replication by setting the following registry value:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\NTDS\Parameters\

Create or modify if exists: Allow Replication With Divergent and Corrupt Partner

DWORD: 1

Once replication has successfully occurred, then set the same value to 0.

If the lingering objects are present because of USN rollback, you should take the far more drastic approach of demoting the DC that was improperly restored and promoting it back into the domain from scratch.  Just point your neuralizer at it and administer the MIB eye exam ala DCPROMO.

Keep in mind however, that you will have to forcibly remove the DC as a normal demotion will fail (see graphic below) since the other DCs refuse to replicate from it. DCPROMO /FORCEREMOVAL will do the job.

Post Op Evaluation

I hope you have learned some DC first aid techniques that will nurse your DCs to their rosy- cheek healthy selves and back in full replication shape. While you are at it, proactively monitor your DCs more closely so that the ailments don’t come back.

Don’t give me your Cooties – When a Domain Controller Refuses to Replicate

If you were to walk onto an airplane during the height of flu season wearing a surgical mask and vehemently hacking and coughing, the guy sitting next to you is likely to scan around for an empty seat to move to, as far away from you as possible. Face it: no one wants your cooties.

A Domain Controller (DC) has a similar aversion to getting sick and when a replication partner is showing signs of illness, the healthy DC doesn’t hesitate to quarantine the sick DC by refusing to replicate from it.

So Doctor, what symptoms does the healthy DC check for before it declares its replication partner, persona non grata?

Symptom Number One – Schema Version Mismatch

The AD Schema is similar to a building code. If I want to build a home in Fort Collins, Larimer County, Colorado where I live, I can’t just march down to Home Depot, load up the truck with supplies, and start framing a home. There are rules to adhere to. So I start off by finding out what the county building code is and then pay for a building permit which legally requires me to adhere to the code. The local code dictates the rules for building the structure - ceilings have to be so high, electrical outlets placed so many feet apart, the roof pitched so many degrees to support the area’s snow load, etc.

Similarly, the AD Schema is the building code for an AD Forest in that all of the Forest’s domains have to abide by the same set of rules for object creation. Each DC in the forest stores a local copy of the AD Schema in its Active Directory database file, loads it into memory and then consults the Schema when new objects are added to the Forest. The Schema basically dictates the types of objects that can be created and what attributes are allowed for those objects, the same way a city’s building code dictates the types of dwellings that can be built and their allowed characteristics.

What would happen if you built your home using an old version of the building code that had an outdated electrical compliance section? Simple - when the home inspector checks your home’s compliance with code, you would be told to pull out all the wiring and start over again; otherwise you are not getting a certificate of occupancy.

In AD, a DC will not replicate data from a partner DC if the partner is using a different version of the building code (schema).  Before replication occurs, the partner DCs will exchange schema version level information. If the DCs don’t have the same schema version, replication of AD Forest Objects between those partners will be held up until the schema itself is fully replicated and both DCs are at the same Schema version.

In reality, this is not a symptom of illness itself as it just means the destination DC will wait to raise its schema level on par with its partner DC, but it does result in a replication delay.

Symptom Number Two – Detection of Lingering Objects

Now this just sounds nasty. Before you consult the medical dictionary, let me first tell you what a lingering object is.

When you create an AD object like a user object, the object will be assigned attributes or properties, such as common name, password, group memberships, etc.  The object then replicates to other DCs in the same domain. AD Replication happens at the attribute level, meaning that when an attribute changes, only that attribute and not the entire object will be replicated.

A little more background is required before you can understand how a DC comes down with a case of lingering objects.

When you delete an object in AD, it becomes a tombstone object and remains in the AD database for a period of time before it is permanently removed. This “obituary” period is known as the tombstone lifetime (TSL) and is how other DCs learn through replication of an object’s demise. The default TSL for Windows 2008 is 180 days although an Enterprise Admin can set it to whatever they want.

Now let’s say that you purposely delete employee Joe’s user object because he just won the lottery and he told you what he really thought of his job before resigning from the company. According to the TSL, 180 days later, object user Joe should be permanently removed from AD.

But what happens if there is a forgotten DC from the domain that has been neglected and has not replicated for 190 days, longer than TSL? That DC never received Joe’s tombstone obituary notice through replication and therefore did not delete its local copy of the user object Joe, maintaining like some Elvis fanatics that user object Joe is still “alive”. A DC not replicating for 190 days sounds like an unlikely scenario, but trust me, it happens.

To continue my scenario, let’s say you send a junior admin who was foolish not to participate in the office lottery pool to the branch office to put things in order. Said Junior Admin fixes the replication problem but also decides out of boredom to run a script that sets the “favorite drink” attribute on all domain user objects to Jolt Cola.

When live-user object Joe’s favorite drink attribute is changed and the branch DC notifies its partner DCs of the change, the other DCs will detect during the replication attempt that the branch DC has a lingering object and will halt inbound replication.

How do they detect this? Because the other DCs will notice that they are trying to replicate “Joe’s favorite drink” attribute for an object they have no knowledge of since they permanently deleted Joe a long time ago. The other DCs are essentially saying: “You are trying to tell me Joe’s favorite drink? – I don’t even know who Joe is!”

Meanwhile flesh and blood Joe is putting his lottery money to work on a beach in Jamaica where his new favorite drink is now a Pina Colada.

Symptom Number Three – Last replication is greater than Tombstone Lifetime

Fortunately, a DC doesn’t have to try and replicate changes to a lingering object to notice something is wrong. A Windows 2008 DC will simply keep track of when their partner DC last replicated and if it is greater than TSL, inbound replication from the loafer DC will be halted. Two error messages will be noticed in the DC’s Directory Service log: Error 1864 and Error 2042.

In the following example, I configured DC1 and DC2 as domain controllers in the same domain. I also manually set the TSL to 2 days and then shutdown DC2. Three days later, I brought DC2 back online, but by failing to replicate longer than TSL, partner DC1 refuses to replicate from it. The following two errors 1864 and 2042 were recorded in DC1’s Directory Service log as can be seen in the following graphics.

 Symptom Number Four – Detection of USN Rollback

Each DC maintains a copy of the AD database file NTDS.DIT and assigns the database a unique ID called the Invocation ID.

At the same time, when a DC creates an object or modifies an object’s attributes in the database, it assigns that LDAP write transaction, a unique locally generated number called an Update Sequence Number (USN).

Each DC maintains its own USN counter and the counter increment is not synchronized with other DCs. Think of the Invocation ID as a Journal and the USN counter as sequential changes to the database recorded in that journal. Each DC maintains its own system of Journals and USNs.

So let’s examine a simple AD database change on DC2 that then replicates to DC1 and see how each DC keeps track of the changes.

Let’s say DC2 starts off the day with a USN counter of 1000 as last recorded in Journal # 1 (its invocation ID). If DC2 then modified three different object attributes at different times of the day, each LDAP write transaction would be sequentially assigned DC2’s next available local USN of 1001, 1002, and 1003 respectively, all recorded in Journal # 1.

DC1 will then replicate those three attributes and record that the last change it received from DC2 was Journal 1/USN 1003.

USN rollback occurs when a DC’s database is improperly restored and “undoes” previous changes in the database, for example, if I restore DC2 from a virtual machine snapshot that was taken right before those three attributes were modified. After the improper restore, DC2 is back to believing its current USN counter is 1000 on Journal # 1, effectively forgetting that it had assigned USNs 1001-1003. It’s partner DC1 however still believes the last update it received from DC2 was Journal 1/USN 1003.

When DC1 detects that it has knowledge of a higher USN for DC2 (Journal 1/USN 1003) than DC2 has for itself (Journal 1/USN 1000), DC1 informs DC2 it is in USN rollback and DC2 halts all inbound and outbound replication and also pauses its NETLOGON service.

This will be recorded in the Directory Service Log on DC2 as errors 2095 and 2103 as seen in the following graphics.

Run, Forest, Run

So now you know why a healthy DC runs in the opposite direction from one that appears ill, but how do you fix one of these problems and nurse the ill DC back to health? Antibiotic therapy and defibrillation steps will be covered in part 2. Stay tuned.

DTR12 Update

The official results of the DTR12 AD Master's class have been published by Ryan Conrad, AD Masters Program Manager at http://blogs.technet.com/b/themasterblog/archive/2011/05/03/mcm-directory-rotation-12-results.aspx

A hearty congratulations goes out to my two fellow classmates Simon and Chris who excelled and achieved the coveted Microsoft Certified Master's designation. To put in perspective their ahievement, they passsed the two very difficult written exams and they survived the grueling nine hour certification lab during the two week class period - the only two students out of fourteen total attending to do this. With just a 15% initial certification rate, Simon and Chris can feel very proud of what they have accomplished.

Although I was happy to pass the two written exams the first time and have that behind me, I was bummed that I did not pass the certification lab and have to wait three months before I can attempt it again. I have already started preparing and have been busy practicing every day in my virtual environment and increasing my consulting engagements to be in the best technical shape possible before July roles around.

To my fellow classmates who are also still in pursuit of the Master's designation, I wish you much luck on exam and lab retakes. Don't give up! Let's meet Simon and Chris at the top of the mountain.

Microsoft Master's Class - A Mind Blowing Experience

This is your brain

This is your brain on Master's

Sunday, April 17, Day 14

Well, the class has finally come to an end. As promised, here's the recap of the last three days. I know you have been riveted to your chair in suspense, wondering about the outcome of the class, or maybe you have better things to do with your time.

On Friday we finished up the day with Group Policy in the morning followed by Disaster Recovery. Manny, one of my fellow classmates volunteered to teach the DR section and delivered an outstanding presentation. Way to go Manny! Then it was off to study on our own for written test # 2, that if you have been keeping up with the blog, you know I miraculously passed on Saturday morning.

Manny teaching DR

Right after we finished written test # 2 and before going to lunch, Ryan sat us down and gave us a twenty minute prep session on what to expect in our Cert lab. The two things that stuck out for me were "don't waste your time memorizing command line syntax" and "if you don't know it by now, you are not going to know it by tomorrow."

We spent Saturday afternoon preparing for Cert lab with these words echoing in our minds. Then all of a sudden, Friday's DR training paid off immediately as we ran into a real disaster. The Hyper-V hosts we were working on all died in the data center, due to a power outage and some Murphy's law reaction by the UPSes.

For us AD MCM candidates, it was an annoyance as we just sat around waiting for the servers to come back up so we could continue our practice session. For the Lync Masters down the hall who were in the middle of their real Cert lab, it was a complete disaster, as the outage happened twice. They were sent home after three weeks without being able to complete their labs. I felt bad for complaining about being stressed after seeing what they had to go through.

I returned to the hotel, making a mental checklist of how I was going to attack the "break and fix" scenarios we would see on the Cert lab: Check the Forest and Domain Functional levels, check schema versions, check replication, check DNS. and on and on and on. I went to bed early that night because as Ryan said (and I believe him most of the time) that if I didn't know it now....

The next morning, we nervously entered into the classroom, wondering if the Data Center staff had put their Disaster Recovery plan in operation and we would be spared the Lync Master's fate. They did and I breathed a sigh of relief, although my relief was short lived as you will understand a little later. Please read on and don't be tempted to stop no matter how appealing that episode of "Dancing with the Stars" is on TV right now. Tivo it.

Ryan handed each person a large white envelope with our name on it and explained that it contained our lab scenarios. We had exactly nine hours to finish.

At 9:15 AM, I opened up the envelope and read through the multi page list of items I needed to work on. After reading through what I was supposed to do, and making a mental calculation about what I could realistically accomplish, I began to triage the scenarios and prioritize my next moves. Unfortunately, my orderly and disciplined approach soon descended into chaos.

You know one of those movies where the world is coming to an end and everything goes wrong and finally the hero saves the day at the last split second? Well, it wasn’t one of those days.  My cert lab world was a friggin’ disaster as I worked as fast as I could through the scenarios yet things weren’t getting fixed fast enough. I took no breaks during the nine hours except to get up, grab a slice of pizza, and sit back down again. I swore often and hit the keys on the keyboard a little harder than I normally do.

In one scenario I tried to use a script to accomplish the requirement but no matter what I did, the script although word for word exactly what was needed, would not work. I remembered something that instructor Mark Cooper had said about Notepad doing some strange things to scripts. I contemplated doing COPY CON (old DOS users know what this means) but ruled it out as the time ticked down. My frustration level went through the roof.

When the lab clock stopped at 6:15 PM, I looked around the room and could tell by my classmates’ frustrated looks that I was not the only one who had trouble getting through the scenarios. We gathered in the hallway and everyone talked about how many lab items they never even got to do because they ran out of time.

Although the mood was somber, slowly it sank in that finally the class was over, and that normal life could resume. Ryan had told us earlier that it would take him some time to adequately grade the labs so we wouldn’t know our results until Tuesday night.  So for now we could just kick back and relax and at least relish the fact that we didn’t crack and call it quits, but stuck it out to the very end. From that point of view, we ALL “passed the test”.

From the classroom we went straight to Bellevue’s world famous Daniel’s Restaurant where Ryan had booked us a private room overlooking the city. Bill Gates’ favorite table was just outside the room, but tonight he wasn’t there. Now it was time to let loose and to shake off all of the stress that had been hanging on us for the last two weeks. Beer, wine and mixed drinks flowed freely and soon enough everyone was smiling, joking and laughing, not to mention eating the best steak I have ever had in my life. You could sense and feel the camaraderie in the room. DTR12 had gone to technical hell and back and we had made it through as a team.

Group pictures were taken. Heartfelt goodbyes were exchanged.  DTR12 then left the building.

So how do I sum up my experience at the Microsoft Master’s class? In one word: REWARDING! I am not talking about the certification, but to the unequaled learning and networking experience that this class offers. I feel that it has taken my technical proficiency to a whole new level and that I also made an awesome group of like minded friends in the process.

Sure, getting the certification is a goal, but as cliché as it may sound, it is always the journey and not the destination that defines our lives, personally or professionally.  Microsoft Master’s class was a positive, sickening, amazing, frustrating, gratifying, mind numbing experience that I heartily recommend to anyone who thinks they are on their technical game. Make the most of your experience if you have the privilege of attending a rotation, and please tell the rat pack of instructors Ryan, Matt, Steve, Mike, and Mark, that DTR12 sent you.

Our view from Daniels restaurant

Our private room at Daniels

De-stressing from the last two weeks

The wine is flowing and we all feel good

Manny and Ryan

Instructor Steve Patrick

DTR12 sans instructors

DTR12 with instructors Steve and Ryan

Saturday, April 16, Day 13

I take back what I said in a previous post. Written exam number one was not the hardest Microsoft exam I have ever taken. Written exam number two now takes that place of honor.

Somehow, the Test Gods smiled down upon me again and I managed to pass.

I don't have time (again) to fill you in on the gory details because tomorrow is Cert lab which promises to be nine more hours in the torture chamber and I have to get ready for being stretched on the rack. I promise, really, that I will write a synopsis of Friday, Saturday, and Sunday when I am on the plane home on Monday. I won't have time tomorrow night after the last exam because the whole class is going out to dinner and celebrating our liberation.

Thursday, April 14, Day 11

I apologize for the non-post yesterday but I was the victim of technology - the hotel's Internet was down. I will fill in the blanks later on today. Stay tuned.

UPDATE!!!!!

The Internet is up but my brain cells are not. Tomorrow is the second written exam and I have migrated to middle earth to my normal study hole, so there will be no news again, until after the test. See you on the other side!

Wednesday, April 13, Day 10

It is very late, I am exhausted, and I need to get some sleep.

Before I drift off............................................................................................................

Sorry, I just dozed off there and now there is drool all over my keyboard.

Before, I head off to la-la land, let me recap the day. Today was PKI, which stands for "Please Keep it Interesting". Fortunately, our new instructor, Mark Cooper, did just that, and it didn't take more than fifteen minutes of his delivery for me to be impressed with his subject knowledge.

I teach PKI in my own classes, and I can tell you from experience, that there are two kinds of people in this world - those that love PKI and those that hate it. I can usually tell which camp the haters are in by their deer in the headlights stare. Normally, when I start talking public-private key pairs, these PKI challengened individuals throw their arms up in despair screaming "What the FEK, we have to learn this stuff?"

Yes Dorothy, you are not in Kansas anymore and PKI is a necessary evil, hence the day and a half that the Master's class spends on it.

Mark knows his stuff when it comes to PKI and I spent the whole day in "knowledge gap fillin' mode" as he connected all the dots for me. Cool! Now I have even more material to drive my students up the certificate wall.

Right after class, as I walked towards the coffee machine for my tenth cup, past the very elaborate lunch buffet the Lync Masters were so nicely treated to, I thought of the upcoming exams.

Our second written exam is on Saturday morning which gives me just two days to be in top shape. It's going to be tough because there is no study day like there was last week. I am beginning to think the Master's instructors actually enjoy inflicting this pain on us. Bunch of rat bast................................................

Oops, drifted off again. What was I saying? Oh yeah, me thinks Matt, Glenn, Ryan, Steve, Mike, and Mark occasionally get together for a beer and plot new ways to make Master candidates squirm. Tomorrow, I am calling Microsoft counseling services to see if there is a Sadism support group that I can refer them to.

Simply no rest for the weary.

Tuesday, April 12, Day 9

Rather than bore you with what we learned in class today (FRS, DFS-Namespace, DFS-Replication), I thought I would spend this blog post talking about the people I have had the privilege to share this experience with so far, my fellow AD Master candidates.

Let me start off by saying that I am blown away by the amount of experience and technical know-how of my classmates. Some of these guys are leaps and bounds beyond me, which I guess you could interpret as meaning their "beanie propellers" spin faster than mine or simply that they are braniacs and techno-whizzes. More importantly they are real, down-to-earth guys. Let me break it down for you.

Simon sits quietly off in the corner near the window and doesn't say much, but when he does, you had better pay attention because it is going to be something significant and very insightful. I put my pen down when he speaks up so I can listen carefully - I can't miss his distinct voice as he is the only one in class speaking the Queen's English.

Right next to Simon is Michael or "Mr. Service Pack" as I have dubbed him because he seems to know when every hotfix, OS tweak and knob was released across all Microsoft product lines. Michael also bears, at least I think, a strong resemblance to a younger Patrick Stewart, the Captain Jean Luc Picard of Star Trek fame. I almost expect Michael to command his computer to "Engage!" as he wonders around the operating system in search of new lifeforms.

Behind Simon is Chris. On the first day of class, I was partnered up with Chris for our very first lab doing AD Schema modifications through a script. Chris commandeered the keyboard, then expertly navigated through the script at lightning speed. I could barely keep up.

In the back row is Brandon, who can really talk tech and who has an uncanny ability to take a highly complex scenario that the class has been arguing about for ten minutes and laying in out in easy to understand terms that sound as smooth as butter.

Next to Brandon is Ambers, our resident comedian. Ambers is a smart techno geek but with a keen sense of humor with impeccable timing and delivery that had me rolling on the floor more than once. If he ever gets burned out on tech, he should consider a stint as a standup comedian.

Across the aisle in the back row is Tyson, also smart as a whip and who isn't afraid to challenge the instructor on a concept that is not clear. Tyson asks the question, and then rephrases the question, and then tweaks the question until the instructor understands the confusion. Thank God for that, because it is through Tyson's questions that every one else who is confused, achieves clarification.

Next to Tyson is Asen. Asen also sits quietly, but when he does ask a question or makes an observation, it is obvious that it has the weight of experience behind it.

In front of Asen is Mark, Mr. PKI, who has been chomping at the bit for Wednesday and Thursday to roll around, the two days we are covering PKI. Mark is an all around smart tech guy and he is a Microsoft PFE for a reason.

Sitting right next to Mark is another PFE - Manny, the consummate tech guy. Manny is the kind of person who has the need to learn how something works in its entirety and he is not satisfied until he does. I can now tell when Manny's curiosity is running in high gear because his eye lids flutter at a fast rate. You had better have your game on when you talk shop with Manny.

Then come Jun who also doesn't talk much but who also knows his stuff. Yun sits right across from me and sometimes I will glance over and catch him scanning through tech article after tech article, while somehow still tracking the conversation in the classroom.

In front of Mark is Matt who seems to have worked on everything and has a tremendous amount of real-world in the trenches experience. When we discuss some obscure technical problem that I have never heard of before, Matt will pipe up and explain how he ran into it and fixed it. I think I will knight him "Sir Been There, Done That."

Finally, we have Martin, quiet and unassuming, you would never know he is a techno-geek because he doesn't look the part. Don't judge a book by its cover - he's been in the field a long time and he has the experience and know how to back it up.

I have spent every night since I got here, holed up in my hotel room with my nose stuck in Technet, trying to keep up with the in-depth class material. I wish there was more time to socialize with these guys because as our first instructor Matt so astutely pointed out on the first day, this experience should not just be about focusing on technology but about building relationships with your classmates. I also have the feeling they would be a fun group to go out drinking with.

I hope that after we part and go our separate ways on Sunday, that I can keep up with everyone going forward and know how they are doing. It has been a real pleasure sharing this experience with them and I wish all of them the best. DTR12 rocks!

Monday, April 11, Day 8

The test gods were good to me. I passed the first exam.

The experience was not pleasant, in fact it was most distasteful. First, there was the pretest anxiety. After  having spent the morning flipping through what seemed like ten million flashcards, trying to tuck away in the back of my mind the one obscure fact that would make or break the exam for me, I was feeling the stress.

Then there was the test itself. Obviously, I can't go into specifics but suffice it to say that it was the hardest Microsoft exam I have ever taken, and I have taken my fair share throughout the years. I'm a visual learner and I can't just read a bunch of text on screen and then scan for the correct answers - I need to "visualize" the scenario first,which requires a lot of diagramming out on paper. I used up every second and barely had enough time to review all the questions.

Since we didn't get our exam scores right away, the class decided to head to "The Commons" and grab some lunch. I was still shaking from the experience and the adrenaline was still pumping through my body. Oh yes, that reminds me. I forgot to mention one thing that made the test experience that much more "exciting".

I take Niacin pills as a natural supplement to control cholesterol and ten minutes into the exam, the pills caused a very unpleasant "flush". I felt my body temperature rise what seemed like ten degrees and I felt like I was going to erupt in flames. The hot flash subsided in about five minutes. Meanwhile I was imagining the next morning 's headlines: "Microsoft candidate spontaneously combusts during test."

The mood at lunch was subdued at times and also filled with nervous laughter. Our exam scores were going to be emailed to us by noon and everyone had their phone out and were constantly checking their inbox. I didn't have my IPhone with me, so I would have to wait until I got back to the classroom to satisfy my curiosity.

By the time we got in the shuttle that would take us back to Building 41, some already knew their scores. Upon returning to the classroom, I grabbed my IPhone from my bag and after reading the good news, I let out a very audible "Yes!" that even the Lync masters down the hall must have heard. I think it was more the adrenaline talking and me feeling a tremendous sense of relief, that the non-stop studying had actually paid off. In a moment, I realized that others were not so lucky and I felt guilty for expressing my joy.

Ryan reminded us that we needed to "reset" after the experience because we were back to a clean slate, meaning that those who were smiling today wouldn't necessarily be smiling on Saturday or Sunday for that matter, when test days rolled around again. Then it was back to the grind.

Surely we will have a short lecture day after such a dreadful test, I thought to myself.

No such luck. It was FRS and DFS-N all afternoon with our new instructor Mike Stephens. As I trudged out of the classroom at 7:30 PM, I could barely hear myself think. I felt like I needed to find the closest diving decompression chamber and spend some time slowly re-acclimating to normal atmosphere. Six more days of this...just six more days...

Today was a victory, but it was too early to celebrate, rather it was time to hunker down again for the next exam.

Don't worry Manny, if you are reading this, drinks are still on me.

Rolling Blackout Due to Overloaded Brain Circuits

Sorry folks, but I don't have time to post yesterday's shenanigans of the MCM class due to time constraints. Tomorrow is the first written exam and I have been glued to a chair studying since I got back to the hotel from class last evening, interrupted only by sleep. My next blog update will be tomorrow AFTER the exam and AFTER I know my score. Wish us all luck! We are going to need it in the "chamber of pain".

HALO Bash - AD Masters vs. Lync Masters

 

The war has started

Ryan and Matt kicking Lync Master butt

Manny and Ambers enjoying a brew

Sensei Glenn and Michael trying out the Fat Tire.



Sensei Ryan and Sensei Glenn defending the honor of AD MCMs.



AD MCM candidates hard at work

Tyson doing a lab with Fat Tire in hand.

Jun and Chris



In the enemy's camp - Lync Masters getting trounced in HALO

Pizza everywhere

The Chamber of Pain



Friday, April 8, Day 5

It's 2 AM and I just finished making a hundred freakin flashcards to study over the weekend, and that was just for the Directory Core Concepts. On Saturday evening and on Sunday, our "day off", I will finish up the rest. On Monday morning, right after the test, I plan on pulling a Mr. Creosote from Monte Python's "The Meaning of Life" and upchucking all the miscellaneous AD trivia swirling around my head. It had better be a big bucket.

On second thought maybe that's not a good idea - I'll probably need it for the hands-on cert lab next week.

So what happened in class today? Well, what I can I say - it was another outrageously exciting day of not seeing the sunlight while glaring at an overhead projector for ten hours. I did find the content a little more interesting today - RODCs and Trust Relationships. Sorry Steve, it's not that I dislike Authentication but it is a little too abstract for me. Now an RODC, that's a real server that I can touch, feel and love on even if it doesn't want to replicate that love back.

Glenn was back at the podium doing his thing. I scanned the room to see what everyone else was up to. Fatigue was obviously setting in as evidenced by the fact that the right side of the room was not asking as many questions as they normally would. The top talkers, Manny and Tyson, were well just a little less talkative today.

Ryan came in to remind us that there was a pizza/beer party and HALO match that evening right after class - AD Master's class against the Lync Master's class that was just down the hall from us. Master against Master.

"Class won't start till 9AM tomorrow," Glenn informed us after the Pizza had arrived. Cool, I thought, that means I can go to bed at 3AM instead of 2 tonight.

The HALO game began, but no one in the classroom was playing. Some students were still doing labs or flipping through their Power Points, while others were hanging out drinking beer and swapping IT war stories. It was obvious that a battle was raging in the game anyway so I guess it was Lync Masters killing Lync Masters. "We have better things to do with our time," joked a fellow student, "Those Lync guys must have an easier class." Slowly the fatigued looks gave way to relaxation and beer smiles.

Finally, Ryan, Matt and Manny jumped into the game. Meanwhile, Tyson was still working on a lab and asked Manny if he knew the syntax for a command. Manny didn't miss a beat, blurting out the answer while gutting a Lync Master at the same time. Multitasking is not a gift, it is a virtue.

Thursday, April 7, Day 4

Another day, another new instructor in this tech paradise called Microsoft Masters.This time it was Steve "Spat" Patrick who was going to give us the low down on Windows Authentication and all of its inner workings. Kerbing my enthusiasm, I slouched down lower in the chair and prepared for the information deluge, and indeed it was. Not that it wasn't interesting stuff, as interesting as Gina and her friends get, but I can only hear so much about ticket this or ticket that before I find myself daydreaming of taking a ride on the Polar Express.

We learned how to troubleshoot and configure Kerberos constrained delegation and protocol transition, and a whole lot of new stuff like how to configure Authentication Mechanism Assurance. It was also cool to see how Steve analyzed authentication related network traces in Network Monitor. "Exercise your curiosity" Steve was fond of saying as he lead us down another rabbit hole to Tech Wonderland. Grinning like the Chesire Cat, he would emerge from the exercise satisfied that he had put yet another technical doubt to bed.

I don't know why but I was under the impression that our first written exam was this Saturday. How did I fat finger that one? I wondered. As it turns out, Ryan gave us the option of either taking the two written tests the same day on the last Saturday of the class or splitting them up and taking the first exam on Monday. Well, that's a no-brainer: If given a choice of falling from a one story or a two story building, which are you going to take?

Some of my classmates worried about failing the first exam and the impact it would have on their mental state and morale going forward. Ryan had a fix for that - he would change the exam so that no one would get their pass/faill result at the end of the exam. Those who wanted to know their results could have their exam score emailed to them the same day and those that did not, would get their results at the end of the course. I am not a patient person when it comes to such matters and chose the instant gratification. Wearing a blindfold in front of an execution squad typically doesn't make the outcome any less painful... I reasoned.

Class ended at 7:30PM, again. I could tell by the look on everyone's face that they were stressed and that I was not the only one feeling the drag. Well, at least I can sleep in on Sunday, I thought to myself, thankful that there was one non-lecture day to look forward to over the horizon. But right now I had to focus on reviewing the lecture items that I didn't have a good grasp on today. Back to the grind.

Wednesday, April 6, Day 3

If this were a regular work week, today would be hump day, but there is no such thing in the life of a Microsoft Master rotation. Instead, it was another day of punishing lecture that lasted from 8AM to 7:30PM, this time dished out by our new instructor Glenn Lecheminant who descended on us like a rat on a cheetoh. Now I can see that Matt's "easy" DNS Case Study from the day before was a psychological ploy to lure us into a sense of false security. That was cruel Matt, very cruel.

You see, Glenn is REPL-MAN, champion of efficient Active Directory replication everywhere and fighter against disjointed databases and lingering objects throughout the free world. Glenn made it clear that no self-respecting IT Professional can don the cape of a Microsoft Master unless they thoroughly understand how AD replication works, how it can be tweaked and most importantly how to troubleshoot it, and it was his job to make sure we did.

OK, I have a little confession to make. I never really understood how AD replication really worked until the last couple of days. Sure I knew the basics, but terms like High Watermark, UptoDatenessVector Table, and InvocationID never seemed to make any sense to me. The light bulb finally came on today in class. I felt embarrassed that I had never properly understood these concepts before, but then again I never had the privilege to hear it explained so authoritatively before either.

This is one of the things I like most about this class. From a learning perspective, you could spend months culling through hundreds of Technet articles, Internet blogs and Microsoft protocol specs to get the deep technical information you need to understand Active Directory replication or you can get it all collated, cooked up, demoed and served nicely to you in the Master's class by an authority like Glenn .

In fact, I would venture to say, that even if you don't give a darn about certifications and acronyms on your business card, attending a Master's class purely from a learning perspective is well worth the money. Of course, everyone in this rotation and I am sure the one's before had as their goal certification, but it is the learning that produces the real value add. I know walking away from this course, the insight and skills I have learned here will help me be a better trainer and a better consultant regardless of the certification outcome.

Another thing I noticed is that the six Microsoft PFEs that are attending this class all really seem to know their stuff. While I am scribbling feverishly to put down on paper a technical concept I just heard of, the PFEs are laid back in their chairs with their arms folded, all knowingly nodding in unison at what the instructor is saying. What do these guys eat for breakfast, Technet Cheerios?

Now, I am back in the hotel room, dreading having to revisit the copious notes from today, all too aware that I am already mentally exhausted and it is only the end of day three. Another pot of coffee is already brewing, Seattle's best of course, and it won't be the last pot tonight. Sleep is just a luxury anyway, right?

Tuesday, April 5, Day 2

When I woke up this morning after less than five hours of sleep, I rubbed my eyes and looked around the room for some duct tape to wrap around my head to prevent yesterday's overload of technical information from leaking out. I looked in the mirror and noticed the bags under my eyes and contemplated taking before and after "Pre and Post Master's course" photos to demonstrate that the grueling event would have a physical impact on my body.

As I walked the short distance to building 40, I mumbled technical acronym after technical acronym to myself like some new age mantra, except I wasn't feeling particularly enlightened but more like in a dazed fog. "Don't forget that one!" I thought. "Dammit, what was it that Matt said we should remember about that one!" One day down, four to go till T1 - Written Test number one.

When I entered the classroom, the same Powerpoint Slide was still on the screen - LDAP. Matt stood at the podium with a sly smile on his face like a drill sergeant glaring down at his raw recruits. "They have no idea" I could almost hear him thinking.

Breathe. Focus. Breathe. Focus.

"Most IT pros don't know LDAP as well as they should" Matt started off saying, but which I heard as, "Get ready, I am about to shove LDAP technical knowledge down your throat until you can do LDAP queries with your eyes closed." Turns out that I was right and we spent the whole morning using LDP.EXE to gingerly get AD to feed back only what we wanted. We also learned how to build custom AD database indexes and measure their performance impact.

"Here's a test tip" Matt said abruptly. All activity in the room ceased as fourteen pairs of eyes focused on him. NDA mode on. "Good to know" I mused. NDA mode off. Then it was off to the technical races again as Matt delved into Client Server interactions of DC Locator.

As I mentioned in yesterday's post, the instruction is phenomenal, but I am not only impressed with the instructor but with my fellow classmates as well. There is a wealth of knowledge and experience in this class and it is refreshing to be able to talk shop with folks whose eyes don't get glazed over with the technical details. One Microsoft PFE is responsible for a HA Active Directory implementation. HA doesn't stand for High Availability but is synonymous with "large".

The afternoon was spent learning all of the intricacies of DNS. If you think you know DNS, you don't. You have to come to a class like this one to really get a feel for all of the nuances that Microsoft has layered on the service.  "I am going to give you a case study to do" Matt said as he divided up the class into two teams. Each team is responsible for coming up with a DNS design for a large corporation and here are the requirements.

Each team huddled to its side of the room and began to plan its implementation. Matt seemed surprised that my team was done so quickly and came over to see if we were goofing off. He then called over the other team, which also had finished quickly, to our side of the room. After I described our well thought out DNS solution, Matt thought carefully and said: "Well, I don't see anything wrong with your approach."

Then we moved to the other side of the room to see how the other team fared and it turned out that their DNS design was identical to ours. Again Matt gave the thumbs up. Both teams broke out in smiles and grunts like the surviving pigs in a game of Angry Birds. Wahoo!!!!!

"Don't celebrate too early" I told myself through my oinks and grunts. The Birds are about to get angrier.