Newest Progress on my XBL data scraper

31. October 2011 17:09 by Cameron in Xbox Live  //  Tags: , , , , , , ,   //   Comments

It's been a while since I last posted, but I have made some great progress on my XBL data scraper. I made some additions to include CDATA tags in the descriptions of achievements as some sample data that I ran into contained ampersand (&) as well as other xml entities in the text and it was creating xml parser errors. Also, I had to add some code to replace ampersand (&) with the xml entity & in xml attributes as I was getting more parser errors with my sample data. This goes the same for other xml entities. Now that these issues have been resolved, I believe my data collector can pull anyone's profile information, given enough time. After testing the scraper with one of my friend's profiles, he has nearly 100 games with 900+ achievements, the initial import into my database took on the order of 7-10 minutes. To deal with manageable updates, I will have to scan a subset of the most recent games played as I don't want it to take as long with incremental updates. The question is how frequently I should do these updates. The incremental updates have not been implemented yet, but will be put into effect soon. I have setup a test site for processing user profiles and viewing collected sample data. I will not make this link public, but if you're interested, shoot me an email and I can share the link with you.

Progress on my Xbox Live data scraper

11. October 2011 13:34 by Cameron in PhantomJS, Xbox Live  //  Tags: , , , , , ,   //   Comments

After many hours working with PhantomJS, I have finally managed to get all available data from xbox.com for my Xbox Live profile using jQuery selectors. The data is outputted as xml and will be imported into my database for quick retrieval and record of historical data. Here is what the xml for my profile looks like:

<xboxGamer>
  <profile gamertag='pcmantinker'>
    <gamerScore>545</gamerScore>
    <avatarSmall>http://avatar.xboxlive.com/avatar/pcmantinker/avatarpic-s.png</avatarSmall>
    <avatarLarge>http://avatar.xboxlive.com/avatar/pcmantinker/avatarpic-l.png</avatarLarge>
    <avatarBody>http://avatar.xboxlive.com/avatar/pcmantinker/avatar-body.png</avatarBody>
    <location>Louisiana</location>
    <bio>Hey guys,

I'm a senior in Computer Science at LSU. I love to play video games, compose piano music, and work on programming projects during my spare time.</bio>
    <motto>Geaux Tigers!</motto>
  </profile>
  <games>
    <game name='DiRT 2'>
      <link>http://marketplace.xbox.com/en-US/Title/1129121824</link>
      <smallImage>http://tiles.xbox.com/consoleAssets/434D0820/en-US/smallboxart.jpg</smallImage>
      <largeImage>http://tiles.xbox.com/consoleAssets/434D0820/en-US/largeboxart.jpg</largeImage>
      <lastPlayed>1/29/2011</lastPlayed>
      <earnedGamerScore>320</earnedGamerScore>
      <totalGamerScore>1000</totalGamerScore>
      <earnedAchievements>21</earnedAchievements>
      <totalAchievements>47</totalAchievements>
      <percentComplete>44</percentComplete>
      <achievementsLink>http://live.xbox.com/en-US/GameCenter/Achievements?titleId=1129121824</achievementsLink>
      <achievements>
        <achievement name='Jet Setter'>
          <description>You've raced in every location. You'll need a new passport soon!</description>
          <image>http://tiles.xbox.com/tiles/Yz/y8/1oCLiGJhbC8wCxsDGyxTWTIwL2FjaC8wLzFEAAAAAOfn5-mTPH8=.jpg</image>
          <gamerScore>15</gamerScore>
          <achievedOn>1/29/2011</achievedOn>
        </achievement>
        <achievement name='All-Star'>
          <description>You're an All-Star driver now!</description>
          <image>http://tiles.xbox.com/tiles/c9/9G/0ICLiGJhbC9CCxsDGyxTWTIwL2FjaC8wLzE2AAAAAOfn5-9p328=.jpg</image>
          <gamerScore>25</gamerScore>
          <achievedOn>1/1/2011</achievedOn>
        </achievement>
        <achievement name="Aren't You Popular!">
          <description>You're pretty popular! You're true friends with four superstars.</description>
          <image>http://tiles.xbox.com/tiles/Pz/3F/14CLiGJhbC9GCxsDGyxTWTIwL2FjaC8wLzEyAAAAAOfn5-jqPSM=.jpg</image>
          <gamerScore>15</gamerScore>
          <achievedOn>1/1/2011</achievedOn>
        </achievement>
        <achievement name='Just Drive!'>
          <description>You drove 100 miles (161 kilometres) without unlocking any other achievement.</description>
          <image>http://tiles.xbox.com/tiles/sP/wW/1oCLiGJhbC9ECBsDGyxTWTIwL2FjaC8wLzIwAAAAAOfn5-k5-Kw=.jpg</image>
          <gamerScore>15</gamerScore>
          <achievedOn>1/1/2011</achievedOn>
        </achievement>
        <achievement name='Terminated!'>
          <description>That's gotta hurt! You took terminal damage at over 140mph (225km/h).</description>
          <image>http://tiles.xbox.com/tiles/Fh/f9/1YCLiGJhbC82FQQcXCdRFzIwL2FjaC8wL0IAAAAA5+fn+tIXDQ==.jpg</image>
          <gamerScore>15</gamerScore>
          <achievedOn>8/14/2010</achievedOn>
        </achievement>
        <achievement name='Big Bucks!'>
          <description>Money! You've earned those Big Bucks!</description>
          <image>http://tiles.xbox.com/tiles/Sh/aE/1ICLiGJhbC9AFQQcXCdRFzIwL2FjaC8wLzQAAAAA5+fn+6sWUQ==.jpg</image>
          <gamerScore>25</gamerScore>
          <achievedOn>8/14/2010</achievedOn>
        </achievement>
        <achievement name='X Games Asia Champ'>
          <description>Banzai! You've won X Games Asia.</description>
          <image>http://tiles.xbox.com/tiles/7P/1v/14CLiGJhbC8yCBsDGyxTWTIwL2FjaC8wLzJGAAAAAOfn5-hA-fA=.jpg</image>
          <gamerScore>25</gamerScore>
          <achievedOn>8/14/2010</achievedOn>
        </achievement>
        <achievement name='Gate Crashed!'>
          <description>Every gate! That's sick! You hit every gate in a Gatecrasher event. </description>
          <image>http://tiles.xbox.com/tiles/Gc/wE/1ICLiGJhbC9ECxsDGyxTWTIwL2FjaC8wLzEwAAAAAOfn5-srzAU=.jpg</image>
          <gamerScore>10</gamerScore>
          <achievedOn>8/14/2010</achievedOn>
        </achievement>
        <achievement name='Friends For Life'>
          <description>You became true friends with a superstar!</description>
          <image>http://tiles.xbox.com/tiles/ij/Tk/1YCLiGJhbC9FCxsDGyxTWTIwL2FjaC8wLzExAAAAAOfn5-rLNJY=.jpg</image>
          <gamerScore>10</gamerScore>
          <achievedOn>8/11/2010</achievedOn>
        </achievement>
        <achievement name='Block Party'>
          <description>You're a Blockbuster. You've beaten Ken in a Throwdown.</description>
          <image>http://tiles.xbox.com/tiles/mt/ce/04CLiGJhbC83CxsDGyxTWTIwL2FjaC8wLzFDAAAAAOfn5-wx14Y=.jpg</image>
          <gamerScore>15</gamerScore>
          <achievedOn>8/10/2010</achievedOn>
        </achievement>
        <achievement name='Piece Of Cake'>
          <description>One challenge down! You've won a Throwdown.</description>
          <image>http://tiles.xbox.com/tiles/4C/em/0YCLiGJhbC9DCxsDGyxTWTIwL2FjaC8wLzE3AAAAAOfn5-6JJ-w=.jpg</image>
          <gamerScore>10</gamerScore>
          <achievedOn>8/10/2010</achievedOn>
        </achievement>
        <achievement name='That All You Got?'>
          <description>You beat a Superstar's time in a Timed Throwdown.</description>
          <image>http://tiles.xbox.com/tiles/1P/RY/04CLiGJhbC9GCRsDGyxTWTIwL2FjaC8wLzMyAAAAAOfn5-x39Mg=.jpg</image>
          <gamerScore>15</gamerScore>
          <achievedOn>8/10/2010</achievedOn>
        </achievement>
        <achievement name='Head Over Wheels'>
          <description>That's some hard charging! You rolled your vehicle but still won the race.</description>
          <image>http://tiles.xbox.com/tiles/8Q/uC/1YCLiGJhbC9MFQQcXCdRFzIwL2FjaC8wLzgAAAAA5+fn+q0L6g==.jpg</image>
          <gamerScore>10</gamerScore>
          <achievedOn>8/10/2010</achievedOn>
        </achievement>
        <achievement name='Getting A Reputation'>
          <description>You're getting a reputation out there. You've reached Level 15.</description>
          <image>http://tiles.xbox.com/tiles/fc/th/04CLiGJhbC9NCxsDGyxTWTIwL2FjaC8wLzE5AAAAAOfn5-xOy2E=.jpg</image>
          <gamerScore>15</gamerScore>
          <achievedOn>8/9/2010</achievedOn>
        </achievement>
        <achievement name='Gut It Out!'>
          <description>It’s the result that counts! You won an event without winning a single race.</description>
          <image>http://tiles.xbox.com/tiles/-x/+l/1oCLiGJhbC9DFQQcXCdRFzIwL2FjaC8wLzcAAAAA5+fn+Yof5A==.jpg</image>
          <gamerScore>10</gamerScore>
          <achievedOn>8/9/2010</achievedOn>
        </achievement>
        <achievement name='Pro'>
          <description>You've proved yourself in the Pro division. Now take it to the next level.</description>
          <image>http://tiles.xbox.com/tiles/xt/Zn/0oCLiGJhbC9BCxsDGyxTWTIwL2FjaC8wLzE1AAAAAOfn5-1I1to=.jpg</image>
          <gamerScore>15</gamerScore>
          <achievedOn>8/9/2010</achievedOn>
        </achievement>
        <achievement name='Keep It Clean!'>
          <description>This game has damage? You won a race without crashing!</description>
          <image>http://tiles.xbox.com/tiles/ox/7c/14CLiGJhbC81FQQcXCdRFzIwL2FjaC8wL0EAAAAA5+fn+PMeuA==.jpg</image>
          <gamerScore>20</gamerScore>
          <achievedOn>8/6/2010</achievedOn>
        </achievement>
        <achievement name='X Games Europe Champ'>
          <description>Blimey! You've won X Games Europe.</description>
          <image>http://tiles.xbox.com/tiles/Wf/RO/1YCLiGJhbC8xCBsDGyxTWTIwL2FjaC8wLzJFAAAAAOfn5-ph9EU=.jpg</image>
          <gamerScore>20</gamerScore>
          <achievedOn>8/5/2010</achievedOn>
        </achievement>
        <achievement name='Rookie Coming Through!'>
          <description>You gotta start somewhere. You've raced in your first DiRT Tour race.</description>
          <image>http://tiles.xbox.com/tiles/VS/6H/04CLiGJhbC9ACxsDGyxTWTIwL2FjaC8wLzE0AAAAAOfn5-yoLkk=.jpg</image>
          <gamerScore>10</gamerScore>
          <achievedOn>6/25/2010</achievedOn>
        </achievement>
        <achievement name='Nailed an Event'>
          <description>You've won an event. Now it's time to take on the world.</description>
          <image>http://tiles.xbox.com/tiles/he/8d/1ICLiGJhbC83FQQcXCdRFzIwL2FjaC8wL0MAAAAA5+fn+zLvng==.jpg</image>
          <gamerScore>10</gamerScore>
          <achievedOn></achievedOn>
        </achievement>
        <achievement name='Gnarly!'>
          <description>Flashy finish! You crossed the finish line with no more than two wheels touching the ground!</description>
          <image>http://tiles.xbox.com/tiles/Yv/Ni/1ICLiGJhbC9NFQQcXCdRFzIwL2FjaC8wLzkAAAAA5+fn+03zeQ==.jpg</image>
          <gamerScore>15</gamerScore>
          <achievedOn></achievedOn>
        </achievement>
        <achievement name='Dedication'>
          <description>Win every event in the DiRT Tour.</description>
          <image>http://tiles.xbox.com/tiles/2b/eV/0jc8P2NhbC9BFQQcXCdRFzIwL2FjaC8wLzUAAAABUFBQ-bq3wg==.jpg</image>
          <gamerScore>80</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='The Full Set'>
          <description>Drive every vehicle in the game.</description>
          <image>http://tiles.xbox.com/tiles/bL/60/0Dc8P2NhbC9CFQQcXCdRFzIwL2FjaC8wLzYAAAABUFBQ-5u+dw==.jpg</image>
          <gamerScore>30</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name="You're The Best, Around!">
          <description>Win a race at every track in the game (in any game mode).</description>
          <image>http://tiles.xbox.com/tiles/fF/1O/1jc8P2NhbC8wFQQcXCdRFzIwL2FjaC8wL0QAAAABUFBQ+WFdZw==.jpg</image>
          <gamerScore>20</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='Dominated!'>
          <description>Win a domination event, getting the maximum available points.</description>
          <image>http://tiles.xbox.com/tiles/76/Wu/1zc8P2NhbC8xFQQcXCdRFzIwL2FjaC8wL0UAAAABUFBQ+IGl9A==.jpg</image>
          <gamerScore>15</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='On The Bubble'>
          <description>Be second to last at each elimination point but go on to win a Last Man Standing race.</description>
          <image>http://tiles.xbox.com/tiles/Wq/yP/1Tc8P2NhbC8yFQQcXCdRFzIwL2FjaC8wL0YAAAABUFBQ+qCsQQ==.jpg</image>
          <gamerScore>15</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='Best Friends Forever!'>
          <description>Become true friends with every superstar.</description>
          <image>http://tiles.xbox.com/tiles/rJ/zU/0Tc8P2NhbC9HCxsDGyxTWTIwL2FjaC8wLzEzAAAAAVBQUP77nLA=.jpg</image>
          <gamerScore>30</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='I like a Challenge'>
          <description>Win 12 Throwdowns.</description>
          <image>http://tiles.xbox.com/tiles/7m/pw/1Tc8P2NhbC9MCxsDGyxTWTIwL2FjaC8wLzE4AAAAAVBQUPpfavI=.jpg</image>
          <gamerScore>20</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='As Good As It Gets'>
          <description>Reach Level 30 in the DiRT Tour.</description>
          <image>http://tiles.xbox.com/tiles/vH/8u/1zc8P2NhbC81CxsDGyxTWTIwL2FjaC8wLzFBAAAAAVBQUPgBf6A=.jpg</image>
          <gamerScore>30</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='Whose Backyard?'>
          <description>Beat Travis Pastrana in a Throwdown.</description>
          <image>http://tiles.xbox.com/tiles/CX/YP/1Tc8P2NhbC82CxsDGyxTWTIwL2FjaC8wLzFCAAAAAVBQUPogdhU=.jpg</image>
          <gamerScore>15</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='World Tour Winner'>
          <description>Win all the World Tour events.</description>
          <image>http://tiles.xbox.com/tiles/8J/2t/0Dc8P2NhbC8xCxsDGyxTWTIwL2FjaC8wLzFFAAAAAVBQUP+Cnew=.jpg</image>
          <gamerScore>50</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='X Games Champ!'>
          <description>Win all the X Games events.</description>
          <image>http://tiles.xbox.com/tiles/RZ/SM/0jc8P2NhbC8yCxsDGyxTWTIwL2FjaC8wLzFGAAAAAVBQUP2jlFk=.jpg</image>
          <gamerScore>50</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='Secret Achievement'>
          <description>Continue playing to unlock this secret achievement.</description>
          <image>/Content/Images/HiddenAchievement.png</image>
          <gamerScore>75</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='Five Of Your Best'>
          <description>Win 5 Pro Tour races over Games for Windows - LIVE.</description>
          <image>http://tiles.xbox.com/tiles/ll/Qm/0jc8P2NhbC9GCBsDGyxTWTIwL2FjaC8wLzIyAAAAAVBQUP0JVIo=.jpg</image>
          <gamerScore>15</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='Tourney'>
          <description>Take part in four tournaments.</description>
          <image>http://tiles.xbox.com/tiles/Ba/zG/0zc8P2NhbC9HCBsDGyxTWTIwL2FjaC8wLzIzAAAAAVBQUPzprBk=.jpg</image>
          <gamerScore>20</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='Famous'>
          <description>Reach Online Fame Level 15.</description>
          <image>http://tiles.xbox.com/tiles/-E/dk/1jc8P2NhbC9ACBsDGyxTWTIwL2FjaC8wLzI0AAAAAVBQUPlLR+A=.jpg</image>
          <gamerScore>20</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='Notorious'>
          <description>Reach Online Fame Level 30.</description>
          <image>http://tiles.xbox.com/tiles/b7/+E/1zc8P2NhbC9BCBsDGyxTWTIwL2FjaC8wLzI1AAAAAVBQUPirv3M=.jpg</image>
          <gamerScore>40</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='Rush Hour'>
          <description>Win a Pro Tour race against 7 competitors over Games for Windows - LIVE.</description>
          <image>http://tiles.xbox.com/tiles/2r/al/1Tc8P2NhbC9CCBsDGyxTWTIwL2FjaC8wLzI2AAAAAVBQUPqKtsY=.jpg</image>
          <gamerScore>10</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='Mix It Up!'>
          <description>Play in every discipline over Games for Windows - LIVE (Pro Tour or Jam Session).</description>
          <image>http://tiles.xbox.com/tiles/SU/5F/1Dc8P2NhbC9DCBsDGyxTWTIwL2FjaC8wLzI3AAAAAVBQUPtqTlU=.jpg</image>
          <gamerScore>15</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='In His Footsteps'>
          <description>Win 25 Rally races in the DiRT Tour.</description>
          <image>http://tiles.xbox.com/tiles/R1/pi/1zc8P2NhbC9MCBsDGyxTWTIwL2FjaC8wLzI4AAAAAVBQUPhNWls=.jpg</image>
          <gamerScore>25</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='With Great Honor'>
          <description>Achieve a 'Cautious' impact rating in Multiplayer races (Pro Tour or Jam Session).</description>
          <image>http://tiles.xbox.com/tiles/1K/KC/1jc8P2NhbC9NCBsDGyxTWTIwL2FjaC8wLzI5AAAAAVBQUPmtosg=.jpg</image>
          <gamerScore>15</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='Mirra Image'>
          <description>Beat Dave Mirra in a Throwdown.</description>
          <image>http://tiles.xbox.com/tiles/FU/88/1Tc8P2NhbC81CBsDGyxTWTIwL2FjaC8wLzJBAAAAAVBQUPoTTwk=.jpg</image>
          <gamerScore>15</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name="I Ain't Afraid Of No Ghost!">
          <description>Choose a ghost, download and beat it!</description>
          <image>http://tiles.xbox.com/tiles/oE/Yd/1zc8P2NhbC82CBsDGyxTWTIwL2FjaC8wLzJCAAAAAVBQUPgyRrw=.jpg</image>
          <gamerScore>10</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='Dialled In'>
          <description>Win a race using a custom vehicle setup.</description>
          <image>http://tiles.xbox.com/tiles/M7/79/1jc8P2NhbC83CBsDGyxTWTIwL2FjaC8wLzJDAAAAAVBQUPnSvi8=.jpg</image>
          <gamerScore>5</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='Two Cups One Girl'>
          <description>Win two team events with the same female team-mate.</description>
          <image>http://tiles.xbox.com/tiles/yl/Vf/0zc8P2NhbC8wCBsDGyxTWTIwL2FjaC8wLzJEAAAAAVBQUPxwVdY=.jpg</image>
          <gamerScore>10</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='X Games America Champ'>
          <description>Win X Games America.</description>
          <image>http://tiles.xbox.com/tiles/8l/xo/1zc8P2NhbC9ECRsDGyxTWTIwL2FjaC8wLzMwAAAAAVBQUPhHXO4=.jpg</image>
          <gamerScore>30</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='Showboating'>
          <description>Bask in your success at X Games America.</description>
          <image>http://tiles.xbox.com/tiles/Ya/SI/1jc8P2NhbC9FCRsDGyxTWTIwL2FjaC8wLzMxAAAAAVBQUPmnpH0=.jpg</image>
          <gamerScore>20</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
      </achievements>
    </game>
    <game name='Halo 2 (PC)'>
      <link>http://marketplace.xbox.com/en-US/Title/1297287183</link>
      <smallImage>http://tiles.xbox.com/consoleAssets/4D53080F/en-US/smallboxart.jpg</smallImage>
      <largeImage>http://tiles.xbox.com/consoleAssets/4D53080F/en-US/largeboxart.jpg</largeImage>
      <lastPlayed>7/3/2010</lastPlayed>
      <earnedGamerScore>225</earnedGamerScore>
      <totalGamerScore>1000</totalGamerScore>
      <earnedAchievements>12</earnedAchievements>
      <totalAchievements>41</totalAchievements>
      <percentComplete>29</percentComplete>
      <achievementsLink>http://live.xbox.com/en-US/GameCenter/Achievements?titleId=1297287183</achievementsLink>
      <achievements>
        <achievement name='Cairo Station'>
          <description>Complete Cairo Station at Normal, Heroic, or Legendary</description>
          <image>http://tiles.xbox.com/tiles/4D/OV/14CLiGJhbC9BFQRrXVBRFzBGL2FjaC8wLzUAAAAA5+fn+Loz+w==.jpg</image>
          <gamerScore>30</gamerScore>
          <achievedOn>7/3/2010</achievedOn>
        </achievement>
        <achievement name='Assassin'>
          <description>Melee an opponent from behind</description>
          <image>http://tiles.xbox.com/tiles/ii/DX/04CLiGJhbC9HFQRrXVBRFzBGL2FjaC8wLzMAAAAA5+fn-PggkQ==.jpg</image>
          <gamerScore>10</gamerScore>
          <achievedOn>9/10/2008</achievedOn>
        </achievement>
        <achievement name='Sniper Kill'>
          <description>Kill an opponent with the sniper rifle or the beam rifle</description>
          <image>http://tiles.xbox.com/tiles/3k/E6/14CLiGJhbC9FCBt0GltTWTBGL2FjaC8wLzIxAAAAAOfn5-gVQcI=.jpg</image>
          <gamerScore>10</gamerScore>
          <achievedOn>9/10/2008</achievedOn>
        </achievement>
        <achievement name='Bonecracker'>
          <description>Hit and kill an opponent with a melee attack</description>
          <image>http://tiles.xbox.com/tiles/c8/t1/1oCLiGJhbC9AFQRrXVBRFzBGL2FjaC8wLzQAAAAA5+fn+VrLaA==.jpg</image>
          <gamerScore>10</gamerScore>
          <achievedOn>9/10/2008</achievedOn>
        </achievement>
        <achievement name='Running Riot'>
          <description>Kill 10 opponents in a row in the same game, without dying</description>
          <image>http://tiles.xbox.com/tiles/nn/lw/1oCLiGJhbC8wCxt0GltTWTBGL2FjaC8wLzFEAAAAAOfn5-lfeYI=.jpg</image>
          <gamerScore>20</gamerScore>
          <achievedOn>9/10/2008</achievedOn>
        </achievement>
        <achievement name='Killing Spree'>
          <description>Kill 5 opponents in a row in the same game, without dying</description>
          <image>http://tiles.xbox.com/tiles/d3/Eo/1YCLiGJhbC9FCxt0GltTWTBGL2FjaC8wLzExAAAAAOfn5-oHcWs=.jpg</image>
          <gamerScore>20</gamerScore>
          <achievedOn>9/10/2008</achievedOn>
        </achievement>
        <achievement name='The Arbiter'>
          <description>Complete The Arbiter at Normal, Heroic, or Legendary</description>
          <image>http://tiles.xbox.com/tiles/Gd/g3/0oCLiGJhbC9GFQRrXVBRFzBGL2FjaC8wLzIAAAAA5+fn-RjYAg==.jpg</image>
          <gamerScore>30</gamerScore>
          <achievedOn>9/5/2008</achievedOn>
        </achievement>
        <achievement name='Stick It'>
          <description>Stick an opponent with a plasma grenade and blow them up</description>
          <image>http://tiles.xbox.com/tiles/a0/gb/1YCLiGJhbC9GCBt0GltTWTBGL2FjaC8wLzIyAAAAAOfn5-o0SHc=.jpg</image>
          <gamerScore>5</gamerScore>
          <achievedOn>9/4/2008</achievedOn>
        </achievement>
        <achievement name='Roadkill'>
          <description>Run over, hit and kill an opponent with a vehicle</description>
          <image>http://tiles.xbox.com/tiles/Z5/LS/04CLiGJhbC83Cxt0GltTWTBGL2FjaC8wLzFDAAAAAOfn5-z9kns=.jpg</image>
          <gamerScore>10</gamerScore>
          <achievedOn>9/4/2008</achievedOn>
        </achievement>
        <achievement name='Flaming Ninja'>
          <description>Kill an opponent that has the Ninja achievement</description>
          <image>http://tiles.xbox.com/tiles/vD/Ls/1oCLiGJhbC83FQRrXVBRFzBGL2FjaC8wL0MAAAAA5+fn+cMypw==.jpg</image>
          <gamerScore>20</gamerScore>
          <achievedOn>9/3/2008</achievedOn>
        </achievement>
        <achievement name='Metropolis'>
          <description>Complete Metropolis at Normal, Heroic, or Legendary</description>
          <image>http://tiles.xbox.com/tiles/O5/Or/0oCLiGJhbC9BCxt0GltTWTBGL2FjaC8wLzE1AAAAAOfn5-2Ekyc=.jpg</image>
          <gamerScore>30</gamerScore>
          <achievedOn>9/3/2008</achievedOn>
        </achievement>
        <achievement name='Outskirts'>
          <description>Complete Outskirts at Normal, Heroic, or Legendary</description>
          <image>http://tiles.xbox.com/tiles/E3/ZN/0oCLiGJhbC9MCxt0GltTWTBGL2FjaC8wLzE4AAAAAOfn5-1idg8=.jpg</image>
          <gamerScore>30</gamerScore>
          <achievedOn>9/3/2008</achievedOn>
        </achievement>
        <achievement name='Oracle'>
          <description>Complete The Oracle at Normal, Heroic, or Legendary</description>
          <image>http://tiles.xbox.com/tiles/HT/ub/1jc8P2NhbC9DCxt0GltTWTBGL2FjaC8wLzE3AAAAAVBQUPm0OwE=.jpg</image>
          <gamerScore>30</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='Delta Halo'>
          <description>Complete Delta Halo at Normal, Heroic, or Legendary</description>
          <image>http://tiles.xbox.com/tiles/W3/di/0Tc8P2NhbC9NFQRrXVBRFzBGL2FjaC8wLzkAAAABUFBQ-k13QA==.jpg</image>
          <gamerScore>30</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='Regret'>
          <description>Complete Regret at Normal, Heroic, or Legendary</description>
          <image>http://tiles.xbox.com/tiles/9D/PD/1Tc8P2NhbC82Cxt0GltTWTBGL2FjaC8wLzFCAAAAAVBQUPrsM+g=.jpg</image>
          <gamerScore>30</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='Sacred Icon'>
          <description>Complete Sacred Icon at Normal, Heroic, or Legendary</description>
          <image>http://tiles.xbox.com/tiles/Dd/hh/0Dc8P2NhbC8xCxt0GltTWTBGL2FjaC8wLzFFAAAAAVBQUP9O2BE=.jpg</image>
          <gamerScore>30</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='Quarantine Zone'>
          <description>Complete Quarantine Zone at Normal, Heroic, or Legendary</description>
          <image>http://tiles.xbox.com/tiles/gN/dc/1Dc8P2NhbC9NCxt0GltTWTBGL2FjaC8wLzE5AAAAAVBQUPtz15w=.jpg</image>
          <gamerScore>30</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='Gravemind'>
          <description>Complete Gravemind at Normal, Heroic, or Legendary</description>
          <image>http://tiles.xbox.com/tiles/1n/hf/1Tc8P2NhbC8xFQRrXVBRFzBGL2FjaC8wL0UAAAABUFBQ+nB4zQ==.jpg</image>
          <gamerScore>30</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='Uprising'>
          <description>Complete Uprising at Normal, Heroic, or Legendary</description>
          <image>http://tiles.xbox.com/tiles/kv/pI/1zc8P2NhbC9BCBt0GltTWTBGL2FjaC8wLzI1AAAAAVBQUPhn+o4=.jpg</image>
          <gamerScore>30</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='High Charity'>
          <description>Complete High Charity at Normal, Heroic, or Legendary</description>
          <image>http://tiles.xbox.com/tiles/Ke/dO/1jc8P2NhbC9NCBt0GltTWTBGL2FjaC8wLzI5AAAAAVBQUPlh5zU=.jpg</image>
          <gamerScore>30</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='The Great Journey'>
          <description>Complete The Great Journey at Normal, Heroic, or Legendary</description>
          <image>http://tiles.xbox.com/tiles/+O/kK/0zc8P2NhbC9HCBt0GltTWTBGL2FjaC8wLzIzAAAAAVBQUPwl6eQ=.jpg</image>
          <gamerScore>30</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='Warrior'>
          <description>Complete the game on Normal difficulty</description>
          <image>http://tiles.xbox.com/tiles/uh/+u/1zc8P2NhbC9MCBt0GltTWTBGL2FjaC8wLzI4AAAAAVBQUPiBH6Y=.jpg</image>
          <gamerScore>100</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='Hero'>
          <description>Complete the game on Heroic difficulty</description>
          <image>http://tiles.xbox.com/tiles/Y3/F+/1zc8P2NhbC8yFQRrXVBRFzBGL2FjaC8wL0YAAAABUFBQ+FFxeA==.jpg</image>
          <gamerScore>50</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='Legend'>
          <description>Complete the game on Legendary difficulty</description>
          <image>http://tiles.xbox.com/tiles/qD/K6/1Dc8P2NhbC9ACxt0GltTWTBGL2FjaC8wLzE0AAAAAVBQUPuVMrQ=.jpg</image>
          <gamerScore>50</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='King of the Scarab'>
          <description>Acquire the Scarab gun on the Metropolis level, any difficulty</description>
          <image>http://tiles.xbox.com/tiles/Ud/kY/0Tc8P2NhbC9HCxt0GltTWTBGL2FjaC8wLzEzAAAAAVBQUP432U0=.jpg</image>
          <gamerScore>20</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='Silent but Deadly'>
          <description>Kill 7 opponents from behind in a row without being spotted, any level, any difficulty</description>
          <image>http://tiles.xbox.com/tiles/uN/FA/0jc8P2NhbC8yCxt0GltTWTBGL2FjaC8wLzFGAAAAAVBQUP1v0aQ=.jpg</image>
          <gamerScore>20</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='Demon'>
          <description>Complete any campaign level without dying once on Normal, Heroic, or Legendary</description>
          <image>http://tiles.xbox.com/tiles/mp/rc/0jc8P2NhbC81FQRrXVBRFzBGL2FjaC8wL0EAAAABUFBQ-fOagQ==.jpg</image>
          <gamerScore>25</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='Go Ape Shiv'>
          <description>Kill an enraged, berserk Brute by melee, any level, any difficulty</description>
          <image>http://tiles.xbox.com/tiles/RY/C-/1Dc8P2NhbC8wFQRrXVBRFzBGL2FjaC8wL0QAAAABUFBQ+5CAXg==.jpg</image>
          <gamerScore>20</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='Counterpoint'>
          <description>Kill the sword carrier</description>
          <image>http://tiles.xbox.com/tiles/xp/ul/0zc8P2NhbC9DFQRrXVBRFzBGL2FjaC8wLzcAAAABUFBQ-Iqb3Q==.jpg</image>
          <gamerScore>5</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='Carjacking'>
          <description>Steal an occupied vehicle from an opponent and take it over</description>
          <image>http://tiles.xbox.com/tiles/VW/NF/0jc8P2NhbC9CFQRrXVBRFzBGL2FjaC8wLzYAAAABUFBQ-WpjTg==.jpg</image>
          <gamerScore>5</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='Violent Cartographer'>
          <description>Play every available default map and stay in each game until it’s over</description>
          <image>http://tiles.xbox.com/tiles/tA/uJ/1Dc8P2NhbC9DCBt0GltTWTBGL2FjaC8wLzI3AAAAAVBQUPumC6g=.jpg</image>
          <gamerScore>25</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='Rainman'>
          <description>Play every available game variant, with at least 3 other players, to the end of each game</description>
          <image>http://tiles.xbox.com/tiles/QT/ri/1zc8P2NhbC81Cxt0GltTWTBGL2FjaC8wLzFBAAAAAVBQUPjNOl0=.jpg</image>
          <gamerScore>25</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='Double Kill'>
          <description>Kill 2 opponents within 4 seconds</description>
          <image>http://tiles.xbox.com/tiles/L5/P9/0Dc8P2NhbC82FQRrXVBRFzBGL2FjaC8wL0IAAAABUFBQ-9KTNA==.jpg</image>
          <gamerScore>20</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='Triple Kill'>
          <description>Kill 3 opponents within 4 seconds</description>
          <image>http://tiles.xbox.com/tiles/AQ/Ko/1jc8P2NhbC9ACBt0GltTWTBGL2FjaC8wLzI0AAAAAVBQUPmHAh0=.jpg</image>
          <gamerScore>20</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='Killtacular'>
          <description>Kill 4 opponents within 4 seconds</description>
          <image>http://tiles.xbox.com/tiles/wi/H4/0Dc8P2NhbC9GCxt0GltTWTBGL2FjaC8wLzEyAAAAAVBQUP-XId4=.jpg</image>
          <gamerScore>20</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='Skewer Stopper'>
          <description>Kill the sword carrier after they kill 5 or more times in a row with the sword</description>
          <image>http://tiles.xbox.com/tiles/Te/Ar/0Tc8P2NhbC9ECBt0GltTWTBGL2FjaC8wLzIwAAAAAVBQUP4E4FE=.jpg</image>
          <gamerScore>20</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='Vigilante'>
          <description>Stop a Killing Spree of another player</description>
          <image>http://tiles.xbox.com/tiles/J-/Np/1Tc8P2NhbC9CCBt0GltTWTBGL2FjaC8wLzI2AAAAAVBQUPpG8zs=.jpg</image>
          <gamerScore>10</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='Air Traffic Control'>
          <description>Blow up the Banshee in flight while manned, with grenades or the rocket launcher</description>
          <image>http://tiles.xbox.com/tiles/rI/jn/1zc8P2NhbC9FFQRrXVBRFzBGL2FjaC8wLzEAAAABUFBQ+MiItw==.jpg</image>
          <gamerScore>20</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='Decorated Soldier'>
          <description>Get awarded at least 8 different medals in one non-team game</description>
          <image>http://tiles.xbox.com/tiles/yI/+C/0Dc8P2NhbC9MFQRrXVBRFzBGL2FjaC8wLzgAAAABUFBQ-62P0w==.jpg</image>
          <gamerScore>10</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='Ninja'>
          <description>In a non-team game, kill 5 people by melee, from behind</description>
          <image>http://tiles.xbox.com/tiles/js/N7/1zc8P2NhbC9CCxt0GltTWTBGL2FjaC8wLzE2AAAAAVBQUPhUw5I=.jpg</image>
          <gamerScore>20</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='Hired Gun'>
          <description>Kill an opponent who has the Legend achievement</description>
          <image>http://tiles.xbox.com/tiles/5N/A5/0zc8P2NhbC9ECxt0GltTWTBGL2FjaC8wLzEwAAAAAVBQUPwW0Pg=.jpg</image>
          <gamerScore>20</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
      </achievements>
    </game>
  </games>
</xboxGamer>

As you can tell, the xml file for a user can be quite large. I've only got two games on Xbox Live currently, but some people may have upwards of 30+ games and each game can have 40+ achievements. In the xml data, there will be some redundancies, but when I import the data into my database, I will make sure to check for uniqueness before adding pre-existing data. The scraper has many phases. First, the raw data import phase where I select the data using jQuery selectors and put the data into javascript objects. The next phase is generating xml from the objects. The last phase is importing the data into my SQL database so that data may be properly related to each user. For example, if two people have the same game, their game lists will simply point to the same game and not make another entry in the database. The same goes for achievements. All unique achievements that exist for each game will be added to the database. For each achievement that a user completes for a game, the system will add an entry that relates the user to the game and achievement.

Month List

Tag cloud