The Solution

“I don’t see it,” Tulsa admitted. “What are you looking at, Sam?”

“Look at how the index is defined, Tulsa. You have a composite key on order_num and item_num, but look at the order of the column names in the index.”

“item_num, order_num. Both are needed to make the key unique. What’s so special about the order of the columns in the key?”

“Everything. Here’s an example. The phone book is essentially indexed by last name, then first name, right? So if I ask you to find Jones, Tulsa, you can do it pretty quick, right?”

“Sure, I just find the Jones’ section, and then look for “Tulsa”. No problem.”

“But what if I asked you to find everyone whose first name is Tulsa? Then how would you do it?”

“Geesh, I’d have to scan the entire phone book, looking at every name, just to see if their first name was Tulsa. The phone book isn’t organized by first name– oh, I get it. The index on the table was created in reverse order, organized by item number first, then order number, so the optimizer couldn’t use the index to find a particular order number. That explains the long delay- the entire 300,000 rows had to be checked. But wait a minute – the last time we ran the query, it completed in less than a second. How could that be?”

“I promised I’d tell you. The clue is that while the logical read count was high, indicating a lot of the table had to be scanned, the physical read count was low, indicating that the second time the query was run, the data was in RAM cache. That’s why you didn’t see this problem all the time- if the data was already in the cache, the query ran in a reasonable amount of time. But if that table got dumped out of the cache, SQL Server had to go to disk to read it back in- resulting in your time-outs.”

Sam and Tulsa checked what had happened during the software upgrade. They discovered that the order_details table had been modified by an upgrade script that created a new table and loaded the data from the existing table into it. The developer who had created the script had inadvertently switched the order of the columns in the PRIMARY CONSTRAINT option.

Sam rebuilt the index on the table, putting order_num first, then item_num, in the PRIMARY KEY constraint. Sam and Tulsa then ran the query again, and this time the optimizer choose the clustered index, resulting in only 7 logical reads to find the data. The agents reported no more problems with the order details window.