Have your say & ask the experts!

Site upgrade - update

9 replies [Last post]
John Duncan's picture
Offline
Joined: 8 Jan 2008
Posts: 22655

Right. Those of you who stayed up late on Thursday to watch the launch of the new site (and our second attempt on Friday morning) are probably wondering what the hell happened, so in a spirit of openness I thought I'd talk you through the issues we faced, what we're doing about them, and when you can expect us to have another go! It really boiled down to three issues - importing the remaining site content which had built up since the last time we did it, the unexpectedly heavy load that was placed on the site the moment we went live, and the supporting infrastructures response to that load. Importing new content We had the site up to date apart from forum posts, which hadn't been updated since about April 20th - about 15,000 new posts altogether. In order to import all of those, we had to make sure that any new users who'd posted since that date were imported, then import the threads, then import all the replies - after that we had to create the URLs for them (the new forum has pretty browser links for each forum thread - like whathifi.com/forum/what-are-you-listening-to instead of community.whathifi.com/forums/t/57127.aspx), make sure they all matched up and were in the right forum, etc etc. We'd anticipated that this was going to take about an hour, but in the end it took about eight - the new user import in particular didn't want to play nice, and for a reason that we didn't get to the bottom of in the time alloted, we lost the turntables and headphones forums altogether. In the end, however, we got there (except for turntables and headphones, which we decided we would go live without and fix the next day), and we readied for launch... Unprecedented Load Just to give you some background, the new site behaves differently from a technical point of view if you're logged in or you're not. If you're not logged in, the site serves up cached HTML pages (provided they've been visited before), and that cache is refreshed at short intervals. Therefore, if nobody's logged in at all, the site doesn't hit the database to get fresh content until it's told to. Conversely, if you are logged in, you get fresh content all the time - practically none of the content is cached and therefore every page you see is a database hit. With that in mind, the most logged in users we normally see is about 20 - the most posts we've ever had in the space of a minute is 21, which whilst not exactly a measure of how many people were logged in at that point, it gives us an idea of our normal authenticated user base. When we went live at around 11pm on Thursday, we saw database activity that suggested we had hundreds of people logging in - ten times our usual activity. As you might expect, the majority of this activity was centred around forum landing pages - the latest posts page and the main forums landing page - and since everybody was logged in (or trying), the pages were all calling the database. Now we did notice that some of those database queries were suboptimal, and needed improving straight away, but we hoped we'd be able to weather the storm, albeit at fairly slow speed, until... Self-Inflicted Hardware Meltdown The new site runs on a public cloud. Which means that, rather than have physical servers that are limited by memory and processing power, we have access to highly scalable hardware which can cope with peaks and troughs in traffic seamlessly. However, this (theoretically) infinitely scalable hardware does have its limits, and when we put our site live, the enormous database load that we suddenly created threatened to bring down lots of other sites on the same cloud, and our provider, as is their right, shut our site down. The reason that we tried twice was that a red herring of a broken server appeared late on Thursday, which made us think that we'd been unlucky so we tried again on Friday. Turned out that the reason the server was broken was probably us! So just after lunch Friday we decided that enough was enough and reverted to the old site, while we licked our wounds (and had a pint)... Now, we've assessed over the weekend and today and think we can address the first two issues above pretty quickly, which kind of make the third one go aay. So we're going to make those changes today and tomorrow, and then hit the site with some heavy load to make sure that it can cope with the activity we saw Thursday and Friday, with a view to going live again towards the end of next week. Any questions, I know you won't be afraid to ask...

__________________

Cambridge Audio StreamMagic 6  |  751BD  |  651A  |  Diamond 9.1 | Minx Xi | Sonos Play:3

Moderator. mail: john.duncan.whf at the mail of g dot com

maxflinn_banned's picture
Offline
Joined: 3 Feb 2009
Posts: 9536
Re: Site upgrade - update

ah , ive got my head around it all now Confused....

i bet you guys had more than one pint ? i needed 7 and im betting it was even more stressful on your end  Stick out tongue

__________________

laptop...airport...marantz pm7001...dynaudio dm 2/7's...

idc
idc's picture
Offline
Joined: 2 Jan 2008
Posts: 7783
Re: Site upgrade - update

But does any of that mean it will work properly for me?

__________________

Internet - laptop - DAC - amp - lots of headphones.

http://idc1966.blogspot.co.uk/

"A music lover will stop what he's doing and stay glued to a favorite piece of music even if it's coming over a 3" speaker or a public-address system..." - Ken Rockwell

 

plastic penguin's picture
Offline
Joined: 28 Apr 2008
Posts: 15601
Re: Site upgrade - update

Crikey, I thought my job was hard....


With the benefit of hindsight, and given the limitations, is there anyway to avoid these blips on subsequent site upgrades?

__________________

Amp: Leema Pulse; Source: Naim CD5i-2, Denon 260MKII, Pro-ject XP I; Speakers: PMC TB2i

 

Formerly known as plastic penguin

Messiah's picture
Offline
Joined: 13 Jan 2008
Posts: 2751
Re: Site upgrade - update

I for one appreciate the feedback JD Yes

Now sort it! Big Smile

__________________

System Photos | Netbook (iTunes) | Onkyo ND-S1 | AVI ADM 9.1 | Yamaha Sub | Epoz Aktimate Mini

 

Andrew Everard's picture
Offline
Joined: 30 May 2007
Posts: 29118
Re: Site upgrade - update
idc:
But does any of that mean it will work properly for me?

No, probably not - but where'd be the fun if it did, huh?

__________________

Audio Editor, Gramophone

ESP2009's picture
Offline
Joined: 16 Feb 2009
Posts: 1789
Re: Site upgrade - update

Interesting.  I (think I) understand how frustrating it all must have been.  I wish you luck this time around.


Incidentally, I am a lazy so-and-so, which means I tend not to logout from one WHF browsing session to the next.  Would this (together with a whole load of other like-minded members) have any significant impact at any time?  I am not necessarily thinking specifically of what you are trying to do with the upgrade.  Should we all make it good practise to logout every time we leave the site, or does it not matter one jot?


Thanks for keeping us informed - it's appreciated.

__________________

Naim Uniti (24/192); ProAc Studio 140 Mk2 ; Synology DS212j & Logitech Squeezebox Touch (it's a kind of magic!) ; Fewer boxes and stringy bits than previously. Subject to change at a whim or WHF delivery.

Andy Clough's picture
Offline
Joined: 27 Apr 2004
Posts: 5709
Re: Site upgrade - update

Now that John has given a comprehensive account of what happened last week, we are of course working on the necessary solutions. That won't happen overnight, but we do intend to have another crack at it - but only once we're confident the issues John mentions have been resolved.

Once again, your patience and understanding is hugely appreciated.

__________________

Andy Clough is Editor-in-Chief of What Hi-Fi? Sound and Vision and whathifi.com

Twitter @AndrewClough

Google+ Andy Clough

nads's picture
Offline
Joined: 29 Nov 2007
Posts: 3203
Re: Site upgrade - update
JohnDuncan:
which kind of make the third one go aay.

So that is the sound a Hardware meltdown makes  Wink

__________________

Not a HiFi Nut. Just has a pile of Nads.

Cofnchtr's picture
Offline
Joined: 4 Oct 2007
Posts: 1383
Re: Site upgrade - update

Hi,

Thanks for the feedback - I needed a few pints just reading it!

Good luck for the next attempt. Fingers etc. crossed it goes well - but then if it did - what would you do with all your free time....angry birds??

Cheers.

Cofnchtr.

EDIT: Meant to ask - is it possible to have a link so that if you're reading a post that's listed on the 2nd, 3rd 4th etc page of the forums, you can get back to the page you came from? You know what I mean? I.E. Within the Home Cinema thread on the 2nd page there's a topic 'Monitor Audio R90's or RS-FX' - if I chose to read this thread in it's entirety I'd read 3 pages. To get back to the 2nd page where I spotted this topic I have to use the back button to take me through 3 pages of thread OR use the direct link at the top of the page to get back to the main Home Cinema forum page, then use the link to get to the 2nd page. If there was a link along the top to take me back to the 2nd page of the thread directly - that'd be a help...if it's not too much trouble....Smile Wink

__________________

Panasonic TX-50GT30B | Pioneer VSX-LX53 | Pioneer BDP-LX54 | B&W M1 x 5 | REL Strata | SKY+HD 1TB | Harmony One Remote | Contentment.