Kickstarter updated the project pages today, which look awesome. However, we had no clue, and we’re totally broken mostly fixed now.
However, here’s the state of things…
(I’ll continue to update this list as I gather more detail)
- Noticed that I was getting more notifications from the cluster than normal that projects were returning invalid data.
- Received emails through the contact form that Kickstarter changed their page layout.
- Suspended acquisition services.
- Looked at the hot list at a couple of the top projects, noticed that some projects that were scraped prior to me noticing were set to some HUGE negative values.
- Rolled back all data for active projects today & flushed the cache so all graphs are back to what they were at midnight until I can verify data being aquired.
- Posted notice in Kicktraq header about the issues.
- Investigating changes to pages
[12:45a EST]
- Pushing new code to test systems
[2:12a EST]
- Test code validated, rolling out to subset of nodes
[3:11a EST]
- node test validated, acquisition rolling out normal updates
- update successful
- Inserting 1am test data to 12am to have daily data for 7/19 (data should only be slightly off for Thursday)
[5:09a EST]
- Ok, I think all is back to normal and all the schedulers are back on their normal schedule.
- I’ve temporarily suspended adding new projects just to be sure. I’ll enable them in a few hours.
- Now, sleep.
Randy Field @ http://ra3ndy.deviantart.com/ graciously allowed us to use his (modified) “sad panda” image. He’s actually running a Kickstarter project @ http://www.kickstarter.com/projects/1345857031/quicksilver-the-great-airship-race that ends in a few hours. You should go check it out!