Xbox Live Operations Blog

Earlier today members were disconnected from Xbox LIVE and found themselves unable to log back in.  This impacted LIVE users on Xbox 360 consoles, those running Zune & Games for Windows LIVE software on PCs, and Windows Phone.

This began just after 9AM Pacific and was resolved roughly two hours later.

The root cause of this outage was human error.  A change resulted in an interruption to normal traffic flow within our datacenters, which logged users out and prevented them from logging back in.  We’re sorry for this, and we’ll be updating our processes to prevent this from happening again in the future. 

If and when issues do occur, we’ll keep you updated through the Status page on Xbox.com and from @XboxSupport on Twitter.

  - e

After a long night of troubleshooting and tweaking by the team, I'm happy to be able to say that your experience on LIVE will be greatly improved today.

Yesterday a number of people were having problems downloading the update, updating the avatars, and re-downloading previously purchased content.  The changes that went in over the last 24 hours have made a world of difference, and we'll continue to watch closely throughout the day.

There were a number of issues raised in the comments of the previous post - we're looking into them, and please keep posting your experiences here as well.

EDIT:

Want to watch ESPN using your 360 on Time Warner Cable? See ESPN's FAQ.

Yesterday we had a handful of reports of in-game freezes - is anyone still seeing that?  If so, what game and at what point?

There's a ton of new functionality in this update including Zune Music, ESPN, improved voice chat quality, enhanced Family Settings, and more.  You can find a more complete listing of what's new on Major Nelson's blog.

As with any significant update like this one, some issues are bound to pop up and we're working hard to resolve them.  In particular, we've seen two intermittent issues today:

Problems downloading the new update or new/updated applications like Zune or Netflix.  There are a couple of different error messages that you might get here, and they all mean essentially the same thing - "Please Try Again".

Being asked to pay for content you want to download that you've previously paid for.  If this happens to you, simply decline and repeat the download process.  This problem is intermittent, and typically on another attempt LIVE should recognize that you already own the content and allow you to re-download it for free.

As we make progress against these issues I'll be back to update this post.  Thanks for your patience.

EDIT: Yes, we aware of the Avatar Editor/Marketplace issue and are working on that as well.

The folks at @XboxSupport have assembled seven tips to help you make sure that you're Reach Ready.  While you're waiting to pick your copy of the game up check them out:

• Get Reach Ready Tip #1: Test your connection to Xbox LIVE for Noble strength. Any problems, go here: http://xbx.lv/9PLdDX

• Get Reach Ready Tip #2: Been NOT twiddling your thumbs waiting for Reach? Check ur controllers batteries for full life!

• Get Reach Ready Tip #3: Your Xbox LIVE Gold subscription’s current -- right?! If unsure check here http://xbx.lv/alsFvp  

• Get Reach Ready Tip #4: Not using an Xbox USB to travel w/ ur Gamertag? Make sure it’s properly recovered: http://xbx.lv/cQbhIG

• Get Reach Ready Tip #5: If you score a Limited Edition Reach console, treat with proper respect- & u might need this: http://xbx.lv/b6nru2

• Get Reach Ready Tip #6: Be ready to party up w/ your friends to Deliver Hope!  Issues, check here: http://xbx.lv/ctI9RR

• Get Reach Ready Tip #7: Follow @XboxSupport for 24-hour Halo: Reach worldwide launch support Sept 13 & 14!

See you on Xbox LIVE!

We’ve opened up the hood on Xbox LIVE and tinkered with how matchmaking works for Modern Warfare 2.  It’s not a dramatic change, and the majority of our users won’t see any difference – but a certain subset of you should be happier now.

When you search for a match, our servers sort through the many thousands of available matches to find the “best” potential games for you.  We’ve changed how we determine “best”, and your country is now an even more significant factor than it was before.  What does this mean to you?

As a gamer, you want to play in a lag-free game.  This means that you need to be playing with gamers who are “near you” on the Internet.  This is why games like MW2 ping potential matches to determine how close they are and reject them if they’re too far away.  If you live somewhere that’s geographically isolated – like Australia, for example – most of the world is not near you.  The speed of light just isn’t fast enough (that’s not something I get to say every day).

On behalf of that Australian gamer we’re now trying harder than ever to find a session in their country for them to join.  Of course, this requires there to be games hosted in Australia with open slots at the moment they’re looking – but we’ve maximized the odds of those gamers getting together.

While we can’t beat the speed of light (yet), we do think that these changes have made finding matches in MW2 a better experience.  If it's helped you out we'd love to hear it!

Later today we’ll be turning off the ability for original Xboxes to connect to Xbox LIVE.  It’s the end of an era, and there’s definitely an odd vibe in the air at Xbox HQ right now.  Re-Volt, Ghost Recon, MechAssault, RTCW, Halo 2 and more – we’ll miss you!

We’ve been getting a slow but steady stream of questions about what this means, so I’m going to answer the most common ones here:

Q: Can I still play my original Xbox games offline?

A: Yes!  This disables support for online multiplayer, leaderboards, marketplace and other online features but does not impact offline play.

Q: Does this impact Halo 2 PC?

A: No!  Halo 2 PC will continue unchanged as it’s a Games for Windows – LIVE game and not for the original Xbox.

Q: When will this take effect?

A: On April 15, 2010 (that's today as I type this).  We aren’t saying exactly when we'll be flipping the switch.

-          e

P.S.  For those of you looking for an update on Matchmaking in Modern Warfare 2 – we are continuing to work on making the experience better for everyone.  There isn’t anything I can say today, but stay tuned.

Starting at 0200 Pacific on April 6th we’ll be rolling a system update out on Xbox LIVE – to see what time that is for you, click here.  It may take a few hours for everyone to be prompted for the update.

This update allows you to use standard USB Flash drives to store profiles, game saves, etc. like you’ve been able to previously on an Xbox 360 Memory Unit.  We’ve also made the process of transferring large amounts of content much simpler with a multi-select option.  If you’re interested in more details – including screenshots – check out Major Nelson’s blog post from March 26th.

First - thanks to everyone who took the time to answer the questions posed in the previous blog entry.  That data was very useful.

We’ve been going over the results with the folks at Infinity Ward, and as a result they’ve made some changes to how the game behaves when it’s looking for a match.  A few of you have already reached out to me to say that you’re seeing an improvement, and that’s great to hear.  We’d love to know what results you get this weekend.

If you’re willing to share your experience again, here’s what we'd like to know – I’ll gather up your results and make sure that IW gets them:

1.       What playlist (TDM/FFA/Stimulus/etc.) you were searching in

2.       How many potential games were found?

3.       How many good games were found?

4.       How long did it take you to find a match from the time you started searching?

If you’re not already, I’d suggest following @fourzerotwo on Twitter to keep an eye out for any information coming directly from IW.  We’ll be here on the blog and manning @XboxSupport as well.

We’ve heard the feedback that some of you are having a hard time finding games in matchmaking with MW2 since Tuesday’s update.  If those of you with this problem can answer a few questions it’ll help us get to the bottom of things.

Please leave a comment and answer the following questions:

1.       Who’s your ISP?

2.       Where (state/country, don’t give your address) do you live?

3.       Describe your networking configuration – router brand, model, etc.

4.       When you run the network troubleshooter in the Xbox Dashboard, what NAT type does it say you have?

5.       When looking for a match, in the top right of the screen it’ll say “x potential games found”. Does x ever go above zero, and if so to what?

6.       When looking for a match, in the top right of the screen it’ll say “x/50 good games”. Does x ever go above zero?

Having this data will be very useful - thanks for helping us help you!

Through a process failure on our part, the Modern Warfare 2 Stimulus Package was released this morning before the Title Update (TU) for MW2 was available.

Unfortunately, the Stimulus Package doesn’t function correctly without the TU.  Once they had been released out of order, it wasn’t a simple thing for us to resolve.  For a couple of hours the result was unpredictable but most people couldn’t get into a game through matchmaking.  In technical terms, it was a big hairball that we had to unwind.

The TU and Matchmaking issues were resolved shortly after 11AM Pacific.  Unfortunately, we’re still working through some billing issues.  These are primarily around the redeeming of POSA cards – POSA stands for “Point of Sale Activation”.  These are the cards that you buy at a retail store that they then activate for you at the checkout register.  Some cards that were properly activated at the retailer are reporting back to us as not having been activated yet.  You haven’t done anything wrong, you won’t lose your money, and you don’t need to go back to the store – just hang in there and try again in a few hours.  The Stimulus Package has been so popular that our POSA vendors that make those magic point cards work are a little backed up.

Other billing features are seeing intermittent problems as well, but at a far lower rate than POSA redemption.  You’re most likely to notice this when trying to redeem a code through Xbox.com, but you may see it on the console as well.  The team has been working all of these issues on your behalf nonstop, and will continue to do so until the last of them are resolved.

As always, keep an eye on the Support page on Xbox.com and the XboxSupport Twitter for the most current information.

Please consider us to have extended the usual “sorry this blog has been quiet for so long” apology.  We know that you’ve read many of them so we’re going to save you from having to read another one here and move on.

While we’ve been gone some pretty amazing things have happened around here.  While much of the world was vacationing between Christmas and New Year’s, you were playing on Xbox LIVE – at one point, more than 2.2 million of you at once!  That was an incredible sight to see.

As I type this we’re preparing for everything that 2010 is going to bring.  If you’re an Xbox LIVE customer you’re in for an great year.  We’ve already seen several excellent games this year, and soon you’ll be getting first crack at new content for Modern Warfare 2.  If you want to take a look at the lineup we’re calling the “Biggest Year in Xbox 360 Gaming History” you can find it here.  Not only do we have all of those amazing titles to look forward to, Project Natal will be hitting store shelves!

If you’re making the journey to PAX East, good for you – we’ll see you there!  A number of members of the Operations team will be in attendance including myself, Stepto, and MacheteBetty.  For the first time we’ll be recording an episode of Major Nelson Radio in front of a live studio audience, Stepto will be giving a talk on Enforcement on Xbox LIVE, and MacheteBetty will be talking about our efforts to make your life easier through @XboxSupport on Twitter.  We’re also looking to schedule a meet-up during the show – more news once we’ve got a location nailed down.

We’re going to be talking to you here more often – so set your RSS reader or follow me on Twitter for updates.

P.S.  While we’re discussing important things, my Gamerscore topped 40,000 yesterday.  That puts me… let me see… carry the one… WAY ahead of Derek and Shreddy.

The Solutions channel is In-dash, and Xbox LIVE is now on Twitter and Facebook!

For those of you who signed up for the Public Preview, you’re already running the latest and greatest version of the Dashboard.  Hopefully you’ve enjoyed the early access, and we’d like to thank you for all the great feedback in the Preview forums on Xbox.com.  Starting tomorrow morning at 2AM Pacific we’ll be rolling it out to all Xbox LIVE users worldwide.

This update has a ton of new functionality – Xbox 360 Games on Demand, Avatar Props and Avatar Marketplace, User Ratings, improved experiences for Parties, Netflix, the Guide, and more!  There’s a fairly comprehensive list of what’s new available at Xbox.com.

Enjoy!

(and if you haven’t checked out Splosion Man yet, I highly recommend it! bort bort bort!)

It’s been a pretty crazy couple of months around here.  I’ll try not to repeat that too often, as I think I’ve been saying that since I joined Xbox back in 2001.  If working in a whirlwind sounds appealing, this is the place you want to be.  Of course, the Madden 10 demo released on Xbox Live this morning so it’s true all over again!

A little over a month ago we made some major upgrades to the datacenters that drive Xbox Live.  We did a number of different things – some of which I can’t talk about – but it’s been made very clear that there’s interest in what we did, so here’s a few tidbits:

Capacity Upgrades.  There’s more than 20 million of you on Xbox Live now, and we need to be prepared for when everyone wants to play the latest and greatest.  With killer titles like Halo: ODST and Modern Warfare 2 shipping this fall we know that you’re all going to be online, and probably all at once!

Deploying New/Improved Features.  In order to enable the cool new functionality that’s coming – Xbox 360 Games on Demand, Avatar Marketplace, User Ratings, and a lot more to say nothing of speeding up the Guide and Account Recovery – we had to do a lot of heavy lifting.  In some cases that’s updating existing bits, but it also meant new services and new hardware.

Adopting New Technology.  Why should you care about this?  As a customer, you don’t have to – but I know that a number of you are interested so I’m mentioning it.  We’ve completed our migration to 64-bit Windows Server 2008 and SQL Server 2008.  This is a platform that delivers some significant advantages for us, which will help us better serve you.

The across-the-board move to 64-bit alone is a big win.  Keeping capacity ahead of demand is a never-ending battle, and easy access to many gigabytes of memory is a blessing when you’re trying to scale.  Windows 2008 means IIS 7, Hyper-V, and a whole host of performance improvements under the hood.  SQL 2008 delivers all sorts of geeky goodness, but I’ll point specifically at the major performance improvements in replication as a wonderful thing.

Starting with the upcoming Public Preview for the next Xbox Live Update you’ll be enjoying the benefits of this work.  There’s still a little time left before we stop taking sign-ups, so if you’re interested in getting access ahead of your 20 million closest friends, click this link!

We’re aware of intermittent errors in Marketplace that are affecting Xbox Live, Xbox.com, and Games for Windows.  You may run into one of a few different status codes including 80004004, 807B01F4, and 8015000a.  It may also show that you haven’t purchased content that you have – don’t worry, we haven’t lost your purchase even though it may appear so.

This can occur whether you’re searching Marketplace from the Dashboard or from within a game.

There’s a team of people actively working on the issue.  Our apologies for the hassle - we’ll get this resolved as soon as we can.

If you have a child account on Xbox Live and you’re from New Zealand you’re probably wondering what’s going on.  This isn’t an error - it’s Xbox Live trying to tell you that we can’t allow your account to download from Marketplace.  See here for more information.

If your account isn't a child account from New Zealand this doesn’t affect you.

It’s good to see millions of you online again!  The vast majority of folks are happily gaming once again, but we’re not done yet.  Here’s an update on a few of the topics that people have been asking about via the blog/email/twitter.

The case of the missing content:

The missing content has been found!  The Sega XBLA titles, Burnout: Paradise Big Surf Island, the latest Rock Band content, and the rest are now available.  Enjoy!

The Optional Media Update:

A number of people have asked about this.  It’s working now – I just verified it myself - but there is a special restriction around it that can be confusing.  You can only download the OMU on a single console per gamertag.  If you need to download on another console – for instance, if you have two 360s  – you’ll need to use a different account on each console.  If you don’t have a second account you can create a new Silver account and that’ll work fine.

Slowness and/or timeouts with http://support.xbox.com:

Some people are having no problems with the site while it’s not working for others.  This is one of our top priorities right now and we’ll get it resolved ASAP.

The Forza 2 Auction House isn’t working:

We’re aware of this and are currently working with the folks at Turn 10 on it.  We’ll get it resolved as soon as we can.

EDIT @ 5:20PM

Comparing Games on Xbox.com:

There have been a number of questions around viewing the achievements of friends on Xbox.com.  When you view someone's profile you'll now see a link for "Compare Games".

There have been a lot of requests for more details about what we did on Tuesday.  Look for a post from me in the next few days on the subject.

Most of the new content has been coming up - will get completed this evening.  Have gotten a lot of feedback from folks who have both been able to download the new arcade content as well as seeing some of the new content they were waiting for appearing.

Not tracking any big issues, so e, Shreddy and I are gonna call it a night.  Folks will still be working around the clock, and they are reading the blog, so if you have any other issues/concerns, please post.

If anything else newsworthy occurs, we will post something in the morning.  Hopefully everyone is having fun playing the new games :)

Got over the immediate marketplace hurdle, and now the issue with the latest arcade titles downloading is resolved.

We still have some newer content we need to get re-added to get it up there, and we are still having issues with the 1 vs. 100 download.

Updates to follow.

Here is what we are tracking and status:

1. Marketplace issues - the issue is understood, however the resolution is complicated.  While I have seen some great suggestions in the blog about how to quickly resolve this, we are having to do this in a rather time intensive way.  I know I am taking a big risk in this crowd by suggesting an ETA, but we are reasonably confident we will have this working within about 2 hours, so by 8:45pm Pacific.
2. Double Purchase - the poor experience related to attempts to re-download these new pieces of content is resolved.  Once issue 1 is fixed, when you attempt to download again, it should show the item as purchased and the option to download again.
3. Achievements comparison between friends on xbox.com - this issue is understood and a fix will be going in tomorrow.

Will send an update when #1 is resolved.  Thanks again for all the updates/feedback.

As e stated, we are tracking and in the process of resolving an interesting issue around Marketplace.  As a part of the work we did yesterday, there were some updates to the tools we use to load content onto the service, and there are some unexpected failures with those new tools that affect new content added to the service in the previous 12-24 hours leading up to the downtime.  That is the only issue of significance we are tracking at this time.

The feedback around issues on this forum has really helped – please keep the feedback coming. 

When the issue is resolved with the new pieces of content, and you are attempting to re-download, please be advised if it appears that you are being charged again, you aren’t.  You can click straight through the messages without being charged.  We have stepped through this ourselves, we know the experience is poor, and are working to see what we can do about it.

For those who offered critical feedback on our performance yesterday, and the fact that this marketplace content issue got by us, I understand and appreciate what you are saying.  Its our job to be flawless, and we clearly didn’t meet the bar yesterday.  I am incredibly proud of the teams for the work that was done, and how hard folks worked to get us where they did – I also appreciate that we work on a service with a demanding customer base; it is as it should be.  I love that part of it.  Please have faith that we are trying really really hard, and are making improvements as fast as we can.

If you are having any other issues, please post.  Thanks again.

The graveyard shift has been investigating and triaging issues all night – thanks for your reports!  We’re working our way though the list and resolving them as quickly as we can.

I want to let you know that we are absolutely aware of the Marketplace issues that people are reporting and that’s at the top of our list.  If you’re not aware, we have a Support page that we keep updated when we run into issues.

If you’re looking to download today’s XBLA releases (M: TG, Sam & Max, and Rocket Riot), Big Surf Island, or other recent pieces of content hang in there.  I’ll let you know as soon as we have those problems resolved.

Its 3:16am here in Redmond... things are looking really good.  Appreciate the feedback that has been sent - we have gone through all issues that were reported and validated fixes for any issues remaining. e just snuck outta here, Shreddy is still barkin' on the conference bridge, and I am thinking its time for me to get a quick nap.  There will be folks monitoring the blog overnight, so again - any issues, please don't hesitate to post.

Thanks for your patience today, and here's to upcoming goodness!