Newest Progress on my XBL data scraper

31. October 2011 17:09 by Cameron in Xbox Live  //  Tags: , , , , , , ,   //   Comments (2)

It's been a while since I last posted, but I have made some great progress on my XBL data scraper. I made some additions to include CDATA tags in the descriptions of achievements as some sample data that I ran into contained ampersand (&) as well as other xml entities in the text and it was creating xml parser errors. Also, I had to add some code to replace ampersand (&) with the xml entity & in xml attributes as I was getting more parser errors with my sample data. This goes the same for other xml entities. Now that these issues have been resolved, I believe my data collector can pull anyone's profile information, given enough time. After testing the scraper with one of my friend's profiles, he has nearly 100 games with 900+ achievements, the initial import into my database took on the order of 7-10 minutes. To deal with manageable updates, I will have to scan a subset of the most recent games played as I don't want it to take as long with incremental updates. The question is how frequently I should do these updates. The incremental updates have not been implemented yet, but will be put into effect soon. I have setup a test site for processing user profiles and viewing collected sample data. I will not make this link public, but if you're interested, shoot me an email and I can share the link with you.

Comments (2) -

Serge Goujon
Serge Goujon
12/2/2011 5:48:26 AM #

Hi. This is nice idea (I'm very interested in too) to build a custom XBL parser but I'd like to ask you what the point of this scraper development? I mean are you planning to share it, sell it or making it just for yourself? (Asking also because I didn't found any alpha/beta code here.)

Cameron
Cameron
12/3/2011 1:41:09 PM #

I eventually plan to license use of my script. It's currently in beta testing, but I hope to have it finished soon. I will keep you posted!

Comments are closed

Month List