My blog has been drifting towards the fun side of my Chicago experience lately, not my job experience. So, for those who don't like that, this post is for you. We'll see which style is the better.
Today I worked with the Keyword Queue. The keyword queue is the crossing of our API interface with Google, Yahoo, and MSN paid search. We throw keyword requests at all of these engines in huge patches, managing bids, managing keyword groups, creating groups, pausing campaigns, the whole nine yards.
Basically, it's the system that links Google, Yahoo, and MSN with the client's campaign so we don't have to log in to each one of them separately. So, what is this keyword queue?
Basically, it's the matrix, kinda. Any request that people put into ConnectCommerce to get changed on the all the respective engines at once gets put into the keyword queue. The queue waits until the unprocessed (newly entered) tasks pile up to about 4000, and then we shoot all these requests over to Google, MSN, and Yahoo. We get charged every time we do this, so we try to keep the requests in pretty large batches.
Unfortunately, not all requests go through without errors. Sometimes people try to edit keywords that don't exist on the engines; sometimes they try to add keywords to groups that don't exist, or move them to places unknown. This is where the errors happen.
If someone does one of these things, it sits in the Keyword Queue (we'll call it the KWQ) marked as an error with a friendly little description. Sometimes the API of the engine just looks at all the incoming data, freaks out, and runs and hides from us and sends us back silly error messages that mean nothing, those get in here too.
When people fix these errors, for instance, by creating a group that needed to be there before they tried to move keywords to that group (which didn't exist earlier) we recycle the tasks. That means, we run a script to say “OK, run these tasks again and see what happens” Lots of times the tasks go through, and happily coexist on both our interface and the engine. Lots of times it doesn't, and it's a whole troubleshooting process in managing the system to get them to go through.
Often times there are 100,000 individual things in the queue, so it's a bit of a nightmare sometimes managing it all. Some specific tasks get resubmitted 9-10 times. That's when we go through a glorified process of zapping these tasks and sending our interface the “Ok, this task has problems, it didn't go through, so, it's canceled, deal with it”
Part of the issue is dealing with the Program Managers who aren't very technically inclined thinkers, who sometimes make simple mistakes repeatedly or don't understand some of the more complex inner workings that hang up the system.
So, it's a balance of dealing with stuff yourself and letting them know the tasks went through OK, or working with the Program Mangers to get it fixed on both ends, and you can just zap the stuff in the queue.
There's a lot more to it, too, we have a syncer that comes along and checks everything in our interface against everything in the Google/MSN/Yahoo API's and makes sure things are up to snuff and the same.
The queue can be accessed through lots of tools, but we have a powerful database-side access for the info. I wrote lots of nifty little scripts to organize the data with access to the backend database so I can diagnose some of the keyword queue problems more effectively. We have an interface that the program managers and us can use to see the keyword queue, but sometimes it's necessary to run advanced queries at the source.
Since recycling and zapping stuff from the queue is sometimes risky business, the program managers don't have access to do those sorts of things. That responsibility lies with us.
That was something that just took up just a couple hours of my day. So, there. I spent the rest of my day writing some documentation for the checking cookies based on our system which allows the program managers to do some analysis themselves.
I also am taking on a pretty big project with Barnes and Noble that needs to verify all of our data files that get sent to them. It needs to make sure there are the right number of ID's, quantities, etc for each order and that all the arithmetic adds up. It needs to process thousands of orders, and if I do a good enough job it might just get stitched into the ConnectCommerce system so we don't have to do it externally.
Where did I go to lunch?
I'm not telling you.
Where did I get a glorious snack?
I'm not telling you that either.
What adventures did I embark on?
That's confidential.
How was my day?
Why should I tell you that?
What were my emotional inner thoughts?
I think you should know the answer to this one now...
How many winning lottery tickets did I have?
If I told you I would have to give you my money.