Gamer Footprint Update

25. February 2015 11:32 by Cameron in Gamer Footprint  //  Tags: , , , ,   //   Comments

It's been a little while since I've updated everyone about Gamer Footprint. Development is picking up with me spending more time each week. I've been primarily working on the backend since the last update, but I have added some functionality to the fronted too.

Users can now register/login using their Google, Microsoft, Facebook, or Twitter account through OAuth2!

Xbox and PSN account linking is now functional and you can see your Xbox Live and PSN presence if you have linked your accounts. Currently, the profile page polls the server every 20 seconds to see if there is a change in presence information. However, I'm looking at making this more like a push notification service from the server through SignalR. The trick will be setting up individual user presence push notifications as it will rely on polling on the server and only pushing to the client when there is a change. I attempted this a little while back, but I couldn't get the connection management quite right to cancel running tasks. I will attempt the push notifications at some point, but it's not a high priority at the moment.

I'm reworking my data persistence and scanning algorithms to update a user's played games on PSN, Xbox and other platforms. Once that is complete, there will be a sort of timeline feature that incorporates all actions that a user wants displayed. Tracking of games played and achievements/trophies won't require a login, but if you want to share your online status with your Gamer Footprint friends and the community, due to privacy settings on each network, you'll need to link your accounts with Gamer Footprint. As of right now, there is no way to view games that you are actively online playing on PSN if you are not a friend of someone. This shouldn't be a big deal though. Rest assured, your information is safe and no passwords for Xbox or PSN accounts are ever stored on our server.

I will be posting more updates in the coming weeks so stay tuned!

Beginning Desktop Client for IGA

2. January 2012 18:45 by Cameron in PSN, Xbox Live  //  Tags: , , , , , , , , , , ,   //   Comments

After some time to think about the most secure method for processing achievements and trophies for an individual user using the user's account credentials, I've decided that using a desktop client rather than server side technology for collecting achievements and trophies is the best approach. Not only is it more secure, but it also reduces the load on my server by turning each user's computer into a node on the IGA network for processing small parts of achievements and trophy data as a whole. The combined data from each user will be uploaded to the IGA dedicated server for importing into the central database. Then data can be viewed from the web interface or smart phone applications.

The web interface and smart phone applications will merely serve as a way to interact with the data from the database rather than pulling data down from the various sources and taking up bandwidth. This is good for smart phone users as data charges can apply for checking for updates frequently.

I'm using a Qt, a cross platform GUI library, for designing the user interface so that the client may run on Windows, Mac OS X, and Linux out of the box. I will post some screen shots as they become available.

Nearly finished with my XBL Scraper and starting my PSN Scraper

3. December 2011 14:31 by Cameron in PhantomJS, Xbox Live, PSN  //  Tags: , , , , , , , , , ,   //   Comments

I've made a ton of progress on my XBL scraper over the past couple of months and am nearing completion. Features include grabbing a user's full list of games and achievements for all time as well as pulling a subset of information based on a date range. I implemented the date range feature because I wanted to speed up incremental updates. One of my beta testers has reported it takes about 15-20 minutes to do a full dump of his information. He also has over 100 games with about 40-60 achievements each.

Right now, my XBL scraper only pulls down information for the currently logged in user. I had thought about having my users on my site provide their login information for pulling down their achievements and recently played games, but there are huge security concerns with me storing Xbox LIVE user names and passwords in my database. One disadvantage to me pulling information from other user profiles is that any achievement that I have not unlocked the image will be returned as a grayscale image. This is not a huge issue though. I'm more leaning towards using my own account to get user data instead of other users' accounts because I don't want to be burdened with security of storing their credentials on my server. If people want color images for achievements, I will just need the community to submit their color images for their achievements to the database. I have also been thinking of a way to have my script run idle on my server while waiting for input from the caller. This will hopefully speed up processing of the data.

On the PSN front, I have begun analyzing various data sources for getting trophy data and recently played games for a PSNID. I've started developing a similar script for pulling information for a PSNID. The PSN scraper is still in its infancy, but it is making steady progress. With my analysis of PSN trophy images, it appears that images for trophies will be returned as color images for public trophy profiles and personal trophy profiles. However, the locked trophies show a locked image next to the trophy. I'm hoping to get community involvement to submit complete trophy images for the locked images. With the way that PSN shows earned trophies, with enough people in the community, the missing images should easily be filled in automatically.

I plan to release premium subscritpion APIs for these scripts in the future. Please stay tuned for more information.

Newest Progress on my XBL data scraper

31. October 2011 17:09 by Cameron in Xbox Live  //  Tags: , , , , , , ,   //   Comments

It's been a while since I last posted, but I have made some great progress on my XBL data scraper. I made some additions to include CDATA tags in the descriptions of achievements as some sample data that I ran into contained ampersand (&) as well as other xml entities in the text and it was creating xml parser errors. Also, I had to add some code to replace ampersand (&) with the xml entity & in xml attributes as I was getting more parser errors with my sample data. This goes the same for other xml entities. Now that these issues have been resolved, I believe my data collector can pull anyone's profile information, given enough time. After testing the scraper with one of my friend's profiles, he has nearly 100 games with 900+ achievements, the initial import into my database took on the order of 7-10 minutes. To deal with manageable updates, I will have to scan a subset of the most recent games played as I don't want it to take as long with incremental updates. The question is how frequently I should do these updates. The incremental updates have not been implemented yet, but will be put into effect soon. I have setup a test site for processing user profiles and viewing collected sample data. I will not make this link public, but if you're interested, shoot me an email and I can share the link with you.

Progress on my Xbox Live data scraper

11. October 2011 13:34 by Cameron in PhantomJS, Xbox Live  //  Tags: , , , , , ,   //   Comments

After many hours working with PhantomJS, I have finally managed to get all available data from xbox.com for my Xbox Live profile using jQuery selectors. The data is outputted as xml and will be imported into my database for quick retrieval and record of historical data. Here is what the xml for my profile looks like:

<xboxGamer>
  <profile gamertag='pcmantinker'>
    <gamerScore>545</gamerScore>
    <avatarSmall>http://avatar.xboxlive.com/avatar/pcmantinker/avatarpic-s.png</avatarSmall>
    <avatarLarge>http://avatar.xboxlive.com/avatar/pcmantinker/avatarpic-l.png</avatarLarge>
    <avatarBody>http://avatar.xboxlive.com/avatar/pcmantinker/avatar-body.png</avatarBody>
    <location>Louisiana</location>
    <bio>Hey guys,

I'm a senior in Computer Science at LSU. I love to play video games, compose piano music, and work on programming projects during my spare time.</bio>
    <motto>Geaux Tigers!</motto>
  </profile>
  <games>
    <game name='DiRT 2'>
      <link>http://marketplace.xbox.com/en-US/Title/1129121824</link>
      <smallImage>http://tiles.xbox.com/consoleAssets/434D0820/en-US/smallboxart.jpg</smallImage>
      <largeImage>http://tiles.xbox.com/consoleAssets/434D0820/en-US/largeboxart.jpg</largeImage>
      <lastPlayed>1/29/2011</lastPlayed>
      <earnedGamerScore>320</earnedGamerScore>
      <totalGamerScore>1000</totalGamerScore>
      <earnedAchievements>21</earnedAchievements>
      <totalAchievements>47</totalAchievements>
      <percentComplete>44</percentComplete>
      <achievementsLink>http://live.xbox.com/en-US/GameCenter/Achievements?titleId=1129121824</achievementsLink>
      <achievements>
        <achievement name='Jet Setter'>
          <description>You've raced in every location. You'll need a new passport soon!</description>
          <image>http://tiles.xbox.com/tiles/Yz/y8/1oCLiGJhbC8wCxsDGyxTWTIwL2FjaC8wLzFEAAAAAOfn5-mTPH8=.jpg</image>
          <gamerScore>15</gamerScore>
          <achievedOn>1/29/2011</achievedOn>
        </achievement>
        <achievement name='All-Star'>
          <description>You're an All-Star driver now!</description>
          <image>http://tiles.xbox.com/tiles/c9/9G/0ICLiGJhbC9CCxsDGyxTWTIwL2FjaC8wLzE2AAAAAOfn5-9p328=.jpg</image>
          <gamerScore>25</gamerScore>
          <achievedOn>1/1/2011</achievedOn>
        </achievement>
        <achievement name="Aren't You Popular!">
          <description>You're pretty popular! You're true friends with four superstars.</description>
          <image>http://tiles.xbox.com/tiles/Pz/3F/14CLiGJhbC9GCxsDGyxTWTIwL2FjaC8wLzEyAAAAAOfn5-jqPSM=.jpg</image>
          <gamerScore>15</gamerScore>
          <achievedOn>1/1/2011</achievedOn>
        </achievement>
        <achievement name='Just Drive!'>
          <description>You drove 100 miles (161 kilometres) without unlocking any other achievement.</description>
          <image>http://tiles.xbox.com/tiles/sP/wW/1oCLiGJhbC9ECBsDGyxTWTIwL2FjaC8wLzIwAAAAAOfn5-k5-Kw=.jpg</image>
          <gamerScore>15</gamerScore>
          <achievedOn>1/1/2011</achievedOn>
        </achievement>
        <achievement name='Terminated!'>
          <description>That's gotta hurt! You took terminal damage at over 140mph (225km/h).</description>
          <image>http://tiles.xbox.com/tiles/Fh/f9/1YCLiGJhbC82FQQcXCdRFzIwL2FjaC8wL0IAAAAA5+fn+tIXDQ==.jpg</image>
          <gamerScore>15</gamerScore>
          <achievedOn>8/14/2010</achievedOn>
        </achievement>
        <achievement name='Big Bucks!'>
          <description>Money! You've earned those Big Bucks!</description>
          <image>http://tiles.xbox.com/tiles/Sh/aE/1ICLiGJhbC9AFQQcXCdRFzIwL2FjaC8wLzQAAAAA5+fn+6sWUQ==.jpg</image>
          <gamerScore>25</gamerScore>
          <achievedOn>8/14/2010</achievedOn>
        </achievement>
        <achievement name='X Games Asia Champ'>
          <description>Banzai! You've won X Games Asia.</description>
          <image>http://tiles.xbox.com/tiles/7P/1v/14CLiGJhbC8yCBsDGyxTWTIwL2FjaC8wLzJGAAAAAOfn5-hA-fA=.jpg</image>
          <gamerScore>25</gamerScore>
          <achievedOn>8/14/2010</achievedOn>
        </achievement>
        <achievement name='Gate Crashed!'>
          <description>Every gate! That's sick! You hit every gate in a Gatecrasher event. </description>
          <image>http://tiles.xbox.com/tiles/Gc/wE/1ICLiGJhbC9ECxsDGyxTWTIwL2FjaC8wLzEwAAAAAOfn5-srzAU=.jpg</image>
          <gamerScore>10</gamerScore>
          <achievedOn>8/14/2010</achievedOn>
        </achievement>
        <achievement name='Friends For Life'>
          <description>You became true friends with a superstar!</description>
          <image>http://tiles.xbox.com/tiles/ij/Tk/1YCLiGJhbC9FCxsDGyxTWTIwL2FjaC8wLzExAAAAAOfn5-rLNJY=.jpg</image>
          <gamerScore>10</gamerScore>
          <achievedOn>8/11/2010</achievedOn>
        </achievement>
        <achievement name='Block Party'>
          <description>You're a Blockbuster. You've beaten Ken in a Throwdown.</description>
          <image>http://tiles.xbox.com/tiles/mt/ce/04CLiGJhbC83CxsDGyxTWTIwL2FjaC8wLzFDAAAAAOfn5-wx14Y=.jpg</image>
          <gamerScore>15</gamerScore>
          <achievedOn>8/10/2010</achievedOn>
        </achievement>
        <achievement name='Piece Of Cake'>
          <description>One challenge down! You've won a Throwdown.</description>
          <image>http://tiles.xbox.com/tiles/4C/em/0YCLiGJhbC9DCxsDGyxTWTIwL2FjaC8wLzE3AAAAAOfn5-6JJ-w=.jpg</image>
          <gamerScore>10</gamerScore>
          <achievedOn>8/10/2010</achievedOn>
        </achievement>
        <achievement name='That All You Got?'>
          <description>You beat a Superstar's time in a Timed Throwdown.</description>
          <image>http://tiles.xbox.com/tiles/1P/RY/04CLiGJhbC9GCRsDGyxTWTIwL2FjaC8wLzMyAAAAAOfn5-x39Mg=.jpg</image>
          <gamerScore>15</gamerScore>
          <achievedOn>8/10/2010</achievedOn>
        </achievement>
        <achievement name='Head Over Wheels'>
          <description>That's some hard charging! You rolled your vehicle but still won the race.</description>
          <image>http://tiles.xbox.com/tiles/8Q/uC/1YCLiGJhbC9MFQQcXCdRFzIwL2FjaC8wLzgAAAAA5+fn+q0L6g==.jpg</image>
          <gamerScore>10</gamerScore>
          <achievedOn>8/10/2010</achievedOn>
        </achievement>
        <achievement name='Getting A Reputation'>
          <description>You're getting a reputation out there. You've reached Level 15.</description>
          <image>http://tiles.xbox.com/tiles/fc/th/04CLiGJhbC9NCxsDGyxTWTIwL2FjaC8wLzE5AAAAAOfn5-xOy2E=.jpg</image>
          <gamerScore>15</gamerScore>
          <achievedOn>8/9/2010</achievedOn>
        </achievement>
        <achievement name='Gut It Out!'>
          <description>It’s the result that counts! You won an event without winning a single race.</description>
          <image>http://tiles.xbox.com/tiles/-x/+l/1oCLiGJhbC9DFQQcXCdRFzIwL2FjaC8wLzcAAAAA5+fn+Yof5A==.jpg</image>
          <gamerScore>10</gamerScore>
          <achievedOn>8/9/2010</achievedOn>
        </achievement>
        <achievement name='Pro'>
          <description>You've proved yourself in the Pro division. Now take it to the next level.</description>
          <image>http://tiles.xbox.com/tiles/xt/Zn/0oCLiGJhbC9BCxsDGyxTWTIwL2FjaC8wLzE1AAAAAOfn5-1I1to=.jpg</image>
          <gamerScore>15</gamerScore>
          <achievedOn>8/9/2010</achievedOn>
        </achievement>
        <achievement name='Keep It Clean!'>
          <description>This game has damage? You won a race without crashing!</description>
          <image>http://tiles.xbox.com/tiles/ox/7c/14CLiGJhbC81FQQcXCdRFzIwL2FjaC8wL0EAAAAA5+fn+PMeuA==.jpg</image>
          <gamerScore>20</gamerScore>
          <achievedOn>8/6/2010</achievedOn>
        </achievement>
        <achievement name='X Games Europe Champ'>
          <description>Blimey! You've won X Games Europe.</description>
          <image>http://tiles.xbox.com/tiles/Wf/RO/1YCLiGJhbC8xCBsDGyxTWTIwL2FjaC8wLzJFAAAAAOfn5-ph9EU=.jpg</image>
          <gamerScore>20</gamerScore>
          <achievedOn>8/5/2010</achievedOn>
        </achievement>
        <achievement name='Rookie Coming Through!'>
          <description>You gotta start somewhere. You've raced in your first DiRT Tour race.</description>
          <image>http://tiles.xbox.com/tiles/VS/6H/04CLiGJhbC9ACxsDGyxTWTIwL2FjaC8wLzE0AAAAAOfn5-yoLkk=.jpg</image>
          <gamerScore>10</gamerScore>
          <achievedOn>6/25/2010</achievedOn>
        </achievement>
        <achievement name='Nailed an Event'>
          <description>You've won an event. Now it's time to take on the world.</description>
          <image>http://tiles.xbox.com/tiles/he/8d/1ICLiGJhbC83FQQcXCdRFzIwL2FjaC8wL0MAAAAA5+fn+zLvng==.jpg</image>
          <gamerScore>10</gamerScore>
          <achievedOn></achievedOn>
        </achievement>
        <achievement name='Gnarly!'>
          <description>Flashy finish! You crossed the finish line with no more than two wheels touching the ground!</description>
          <image>http://tiles.xbox.com/tiles/Yv/Ni/1ICLiGJhbC9NFQQcXCdRFzIwL2FjaC8wLzkAAAAA5+fn+03zeQ==.jpg</image>
          <gamerScore>15</gamerScore>
          <achievedOn></achievedOn>
        </achievement>
        <achievement name='Dedication'>
          <description>Win every event in the DiRT Tour.</description>
          <image>http://tiles.xbox.com/tiles/2b/eV/0jc8P2NhbC9BFQQcXCdRFzIwL2FjaC8wLzUAAAABUFBQ-bq3wg==.jpg</image>
          <gamerScore>80</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='The Full Set'>
          <description>Drive every vehicle in the game.</description>
          <image>http://tiles.xbox.com/tiles/bL/60/0Dc8P2NhbC9CFQQcXCdRFzIwL2FjaC8wLzYAAAABUFBQ-5u+dw==.jpg</image>
          <gamerScore>30</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name="You're The Best, Around!">
          <description>Win a race at every track in the game (in any game mode).</description>
          <image>http://tiles.xbox.com/tiles/fF/1O/1jc8P2NhbC8wFQQcXCdRFzIwL2FjaC8wL0QAAAABUFBQ+WFdZw==.jpg</image>
          <gamerScore>20</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='Dominated!'>
          <description>Win a domination event, getting the maximum available points.</description>
          <image>http://tiles.xbox.com/tiles/76/Wu/1zc8P2NhbC8xFQQcXCdRFzIwL2FjaC8wL0UAAAABUFBQ+IGl9A==.jpg</image>
          <gamerScore>15</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='On The Bubble'>
          <description>Be second to last at each elimination point but go on to win a Last Man Standing race.</description>
          <image>http://tiles.xbox.com/tiles/Wq/yP/1Tc8P2NhbC8yFQQcXCdRFzIwL2FjaC8wL0YAAAABUFBQ+qCsQQ==.jpg</image>
          <gamerScore>15</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='Best Friends Forever!'>
          <description>Become true friends with every superstar.</description>
          <image>http://tiles.xbox.com/tiles/rJ/zU/0Tc8P2NhbC9HCxsDGyxTWTIwL2FjaC8wLzEzAAAAAVBQUP77nLA=.jpg</image>
          <gamerScore>30</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='I like a Challenge'>
          <description>Win 12 Throwdowns.</description>
          <image>http://tiles.xbox.com/tiles/7m/pw/1Tc8P2NhbC9MCxsDGyxTWTIwL2FjaC8wLzE4AAAAAVBQUPpfavI=.jpg</image>
          <gamerScore>20</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='As Good As It Gets'>
          <description>Reach Level 30 in the DiRT Tour.</description>
          <image>http://tiles.xbox.com/tiles/vH/8u/1zc8P2NhbC81CxsDGyxTWTIwL2FjaC8wLzFBAAAAAVBQUPgBf6A=.jpg</image>
          <gamerScore>30</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='Whose Backyard?'>
          <description>Beat Travis Pastrana in a Throwdown.</description>
          <image>http://tiles.xbox.com/tiles/CX/YP/1Tc8P2NhbC82CxsDGyxTWTIwL2FjaC8wLzFCAAAAAVBQUPogdhU=.jpg</image>
          <gamerScore>15</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='World Tour Winner'>
          <description>Win all the World Tour events.</description>
          <image>http://tiles.xbox.com/tiles/8J/2t/0Dc8P2NhbC8xCxsDGyxTWTIwL2FjaC8wLzFFAAAAAVBQUP+Cnew=.jpg</image>
          <gamerScore>50</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='X Games Champ!'>
          <description>Win all the X Games events.</description>
          <image>http://tiles.xbox.com/tiles/RZ/SM/0jc8P2NhbC8yCxsDGyxTWTIwL2FjaC8wLzFGAAAAAVBQUP2jlFk=.jpg</image>
          <gamerScore>50</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='Secret Achievement'>
          <description>Continue playing to unlock this secret achievement.</description>
          <image>/Content/Images/HiddenAchievement.png</image>
          <gamerScore>75</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='Five Of Your Best'>
          <description>Win 5 Pro Tour races over Games for Windows - LIVE.</description>
          <image>http://tiles.xbox.com/tiles/ll/Qm/0jc8P2NhbC9GCBsDGyxTWTIwL2FjaC8wLzIyAAAAAVBQUP0JVIo=.jpg</image>
          <gamerScore>15</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='Tourney'>
          <description>Take part in four tournaments.</description>
          <image>http://tiles.xbox.com/tiles/Ba/zG/0zc8P2NhbC9HCBsDGyxTWTIwL2FjaC8wLzIzAAAAAVBQUPzprBk=.jpg</image>
          <gamerScore>20</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='Famous'>
          <description>Reach Online Fame Level 15.</description>
          <image>http://tiles.xbox.com/tiles/-E/dk/1jc8P2NhbC9ACBsDGyxTWTIwL2FjaC8wLzI0AAAAAVBQUPlLR+A=.jpg</image>
          <gamerScore>20</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='Notorious'>
          <description>Reach Online Fame Level 30.</description>
          <image>http://tiles.xbox.com/tiles/b7/+E/1zc8P2NhbC9BCBsDGyxTWTIwL2FjaC8wLzI1AAAAAVBQUPirv3M=.jpg</image>
          <gamerScore>40</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='Rush Hour'>
          <description>Win a Pro Tour race against 7 competitors over Games for Windows - LIVE.</description>
          <image>http://tiles.xbox.com/tiles/2r/al/1Tc8P2NhbC9CCBsDGyxTWTIwL2FjaC8wLzI2AAAAAVBQUPqKtsY=.jpg</image>
          <gamerScore>10</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='Mix It Up!'>
          <description>Play in every discipline over Games for Windows - LIVE (Pro Tour or Jam Session).</description>
          <image>http://tiles.xbox.com/tiles/SU/5F/1Dc8P2NhbC9DCBsDGyxTWTIwL2FjaC8wLzI3AAAAAVBQUPtqTlU=.jpg</image>
          <gamerScore>15</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='In His Footsteps'>
          <description>Win 25 Rally races in the DiRT Tour.</description>
          <image>http://tiles.xbox.com/tiles/R1/pi/1zc8P2NhbC9MCBsDGyxTWTIwL2FjaC8wLzI4AAAAAVBQUPhNWls=.jpg</image>
          <gamerScore>25</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='With Great Honor'>
          <description>Achieve a 'Cautious' impact rating in Multiplayer races (Pro Tour or Jam Session).</description>
          <image>http://tiles.xbox.com/tiles/1K/KC/1jc8P2NhbC9NCBsDGyxTWTIwL2FjaC8wLzI5AAAAAVBQUPmtosg=.jpg</image>
          <gamerScore>15</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='Mirra Image'>
          <description>Beat Dave Mirra in a Throwdown.</description>
          <image>http://tiles.xbox.com/tiles/FU/88/1Tc8P2NhbC81CBsDGyxTWTIwL2FjaC8wLzJBAAAAAVBQUPoTTwk=.jpg</image>
          <gamerScore>15</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name="I Ain't Afraid Of No Ghost!">
          <description>Choose a ghost, download and beat it!</description>
          <image>http://tiles.xbox.com/tiles/oE/Yd/1zc8P2NhbC82CBsDGyxTWTIwL2FjaC8wLzJCAAAAAVBQUPgyRrw=.jpg</image>
          <gamerScore>10</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='Dialled In'>
          <description>Win a race using a custom vehicle setup.</description>
          <image>http://tiles.xbox.com/tiles/M7/79/1jc8P2NhbC83CBsDGyxTWTIwL2FjaC8wLzJDAAAAAVBQUPnSvi8=.jpg</image>
          <gamerScore>5</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='Two Cups One Girl'>
          <description>Win two team events with the same female team-mate.</description>
          <image>http://tiles.xbox.com/tiles/yl/Vf/0zc8P2NhbC8wCBsDGyxTWTIwL2FjaC8wLzJEAAAAAVBQUPxwVdY=.jpg</image>
          <gamerScore>10</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='X Games America Champ'>
          <description>Win X Games America.</description>
          <image>http://tiles.xbox.com/tiles/8l/xo/1zc8P2NhbC9ECRsDGyxTWTIwL2FjaC8wLzMwAAAAAVBQUPhHXO4=.jpg</image>
          <gamerScore>30</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='Showboating'>
          <description>Bask in your success at X Games America.</description>
          <image>http://tiles.xbox.com/tiles/Ya/SI/1jc8P2NhbC9FCRsDGyxTWTIwL2FjaC8wLzMxAAAAAVBQUPmnpH0=.jpg</image>
          <gamerScore>20</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
      </achievements>
    </game>
    <game name='Halo 2 (PC)'>
      <link>http://marketplace.xbox.com/en-US/Title/1297287183</link>
      <smallImage>http://tiles.xbox.com/consoleAssets/4D53080F/en-US/smallboxart.jpg</smallImage>
      <largeImage>http://tiles.xbox.com/consoleAssets/4D53080F/en-US/largeboxart.jpg</largeImage>
      <lastPlayed>7/3/2010</lastPlayed>
      <earnedGamerScore>225</earnedGamerScore>
      <totalGamerScore>1000</totalGamerScore>
      <earnedAchievements>12</earnedAchievements>
      <totalAchievements>41</totalAchievements>
      <percentComplete>29</percentComplete>
      <achievementsLink>http://live.xbox.com/en-US/GameCenter/Achievements?titleId=1297287183</achievementsLink>
      <achievements>
        <achievement name='Cairo Station'>
          <description>Complete Cairo Station at Normal, Heroic, or Legendary</description>
          <image>http://tiles.xbox.com/tiles/4D/OV/14CLiGJhbC9BFQRrXVBRFzBGL2FjaC8wLzUAAAAA5+fn+Loz+w==.jpg</image>
          <gamerScore>30</gamerScore>
          <achievedOn>7/3/2010</achievedOn>
        </achievement>
        <achievement name='Assassin'>
          <description>Melee an opponent from behind</description>
          <image>http://tiles.xbox.com/tiles/ii/DX/04CLiGJhbC9HFQRrXVBRFzBGL2FjaC8wLzMAAAAA5+fn-PggkQ==.jpg</image>
          <gamerScore>10</gamerScore>
          <achievedOn>9/10/2008</achievedOn>
        </achievement>
        <achievement name='Sniper Kill'>
          <description>Kill an opponent with the sniper rifle or the beam rifle</description>
          <image>http://tiles.xbox.com/tiles/3k/E6/14CLiGJhbC9FCBt0GltTWTBGL2FjaC8wLzIxAAAAAOfn5-gVQcI=.jpg</image>
          <gamerScore>10</gamerScore>
          <achievedOn>9/10/2008</achievedOn>
        </achievement>
        <achievement name='Bonecracker'>
          <description>Hit and kill an opponent with a melee attack</description>
          <image>http://tiles.xbox.com/tiles/c8/t1/1oCLiGJhbC9AFQRrXVBRFzBGL2FjaC8wLzQAAAAA5+fn+VrLaA==.jpg</image>
          <gamerScore>10</gamerScore>
          <achievedOn>9/10/2008</achievedOn>
        </achievement>
        <achievement name='Running Riot'>
          <description>Kill 10 opponents in a row in the same game, without dying</description>
          <image>http://tiles.xbox.com/tiles/nn/lw/1oCLiGJhbC8wCxt0GltTWTBGL2FjaC8wLzFEAAAAAOfn5-lfeYI=.jpg</image>
          <gamerScore>20</gamerScore>
          <achievedOn>9/10/2008</achievedOn>
        </achievement>
        <achievement name='Killing Spree'>
          <description>Kill 5 opponents in a row in the same game, without dying</description>
          <image>http://tiles.xbox.com/tiles/d3/Eo/1YCLiGJhbC9FCxt0GltTWTBGL2FjaC8wLzExAAAAAOfn5-oHcWs=.jpg</image>
          <gamerScore>20</gamerScore>
          <achievedOn>9/10/2008</achievedOn>
        </achievement>
        <achievement name='The Arbiter'>
          <description>Complete The Arbiter at Normal, Heroic, or Legendary</description>
          <image>http://tiles.xbox.com/tiles/Gd/g3/0oCLiGJhbC9GFQRrXVBRFzBGL2FjaC8wLzIAAAAA5+fn-RjYAg==.jpg</image>
          <gamerScore>30</gamerScore>
          <achievedOn>9/5/2008</achievedOn>
        </achievement>
        <achievement name='Stick It'>
          <description>Stick an opponent with a plasma grenade and blow them up</description>
          <image>http://tiles.xbox.com/tiles/a0/gb/1YCLiGJhbC9GCBt0GltTWTBGL2FjaC8wLzIyAAAAAOfn5-o0SHc=.jpg</image>
          <gamerScore>5</gamerScore>
          <achievedOn>9/4/2008</achievedOn>
        </achievement>
        <achievement name='Roadkill'>
          <description>Run over, hit and kill an opponent with a vehicle</description>
          <image>http://tiles.xbox.com/tiles/Z5/LS/04CLiGJhbC83Cxt0GltTWTBGL2FjaC8wLzFDAAAAAOfn5-z9kns=.jpg</image>
          <gamerScore>10</gamerScore>
          <achievedOn>9/4/2008</achievedOn>
        </achievement>
        <achievement name='Flaming Ninja'>
          <description>Kill an opponent that has the Ninja achievement</description>
          <image>http://tiles.xbox.com/tiles/vD/Ls/1oCLiGJhbC83FQRrXVBRFzBGL2FjaC8wL0MAAAAA5+fn+cMypw==.jpg</image>
          <gamerScore>20</gamerScore>
          <achievedOn>9/3/2008</achievedOn>
        </achievement>
        <achievement name='Metropolis'>
          <description>Complete Metropolis at Normal, Heroic, or Legendary</description>
          <image>http://tiles.xbox.com/tiles/O5/Or/0oCLiGJhbC9BCxt0GltTWTBGL2FjaC8wLzE1AAAAAOfn5-2Ekyc=.jpg</image>
          <gamerScore>30</gamerScore>
          <achievedOn>9/3/2008</achievedOn>
        </achievement>
        <achievement name='Outskirts'>
          <description>Complete Outskirts at Normal, Heroic, or Legendary</description>
          <image>http://tiles.xbox.com/tiles/E3/ZN/0oCLiGJhbC9MCxt0GltTWTBGL2FjaC8wLzE4AAAAAOfn5-1idg8=.jpg</image>
          <gamerScore>30</gamerScore>
          <achievedOn>9/3/2008</achievedOn>
        </achievement>
        <achievement name='Oracle'>
          <description>Complete The Oracle at Normal, Heroic, or Legendary</description>
          <image>http://tiles.xbox.com/tiles/HT/ub/1jc8P2NhbC9DCxt0GltTWTBGL2FjaC8wLzE3AAAAAVBQUPm0OwE=.jpg</image>
          <gamerScore>30</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='Delta Halo'>
          <description>Complete Delta Halo at Normal, Heroic, or Legendary</description>
          <image>http://tiles.xbox.com/tiles/W3/di/0Tc8P2NhbC9NFQRrXVBRFzBGL2FjaC8wLzkAAAABUFBQ-k13QA==.jpg</image>
          <gamerScore>30</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='Regret'>
          <description>Complete Regret at Normal, Heroic, or Legendary</description>
          <image>http://tiles.xbox.com/tiles/9D/PD/1Tc8P2NhbC82Cxt0GltTWTBGL2FjaC8wLzFCAAAAAVBQUPrsM+g=.jpg</image>
          <gamerScore>30</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='Sacred Icon'>
          <description>Complete Sacred Icon at Normal, Heroic, or Legendary</description>
          <image>http://tiles.xbox.com/tiles/Dd/hh/0Dc8P2NhbC8xCxt0GltTWTBGL2FjaC8wLzFFAAAAAVBQUP9O2BE=.jpg</image>
          <gamerScore>30</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='Quarantine Zone'>
          <description>Complete Quarantine Zone at Normal, Heroic, or Legendary</description>
          <image>http://tiles.xbox.com/tiles/gN/dc/1Dc8P2NhbC9NCxt0GltTWTBGL2FjaC8wLzE5AAAAAVBQUPtz15w=.jpg</image>
          <gamerScore>30</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='Gravemind'>
          <description>Complete Gravemind at Normal, Heroic, or Legendary</description>
          <image>http://tiles.xbox.com/tiles/1n/hf/1Tc8P2NhbC8xFQRrXVBRFzBGL2FjaC8wL0UAAAABUFBQ+nB4zQ==.jpg</image>
          <gamerScore>30</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='Uprising'>
          <description>Complete Uprising at Normal, Heroic, or Legendary</description>
          <image>http://tiles.xbox.com/tiles/kv/pI/1zc8P2NhbC9BCBt0GltTWTBGL2FjaC8wLzI1AAAAAVBQUPhn+o4=.jpg</image>
          <gamerScore>30</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='High Charity'>
          <description>Complete High Charity at Normal, Heroic, or Legendary</description>
          <image>http://tiles.xbox.com/tiles/Ke/dO/1jc8P2NhbC9NCBt0GltTWTBGL2FjaC8wLzI5AAAAAVBQUPlh5zU=.jpg</image>
          <gamerScore>30</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='The Great Journey'>
          <description>Complete The Great Journey at Normal, Heroic, or Legendary</description>
          <image>http://tiles.xbox.com/tiles/+O/kK/0zc8P2NhbC9HCBt0GltTWTBGL2FjaC8wLzIzAAAAAVBQUPwl6eQ=.jpg</image>
          <gamerScore>30</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='Warrior'>
          <description>Complete the game on Normal difficulty</description>
          <image>http://tiles.xbox.com/tiles/uh/+u/1zc8P2NhbC9MCBt0GltTWTBGL2FjaC8wLzI4AAAAAVBQUPiBH6Y=.jpg</image>
          <gamerScore>100</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='Hero'>
          <description>Complete the game on Heroic difficulty</description>
          <image>http://tiles.xbox.com/tiles/Y3/F+/1zc8P2NhbC8yFQRrXVBRFzBGL2FjaC8wL0YAAAABUFBQ+FFxeA==.jpg</image>
          <gamerScore>50</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='Legend'>
          <description>Complete the game on Legendary difficulty</description>
          <image>http://tiles.xbox.com/tiles/qD/K6/1Dc8P2NhbC9ACxt0GltTWTBGL2FjaC8wLzE0AAAAAVBQUPuVMrQ=.jpg</image>
          <gamerScore>50</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='King of the Scarab'>
          <description>Acquire the Scarab gun on the Metropolis level, any difficulty</description>
          <image>http://tiles.xbox.com/tiles/Ud/kY/0Tc8P2NhbC9HCxt0GltTWTBGL2FjaC8wLzEzAAAAAVBQUP432U0=.jpg</image>
          <gamerScore>20</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='Silent but Deadly'>
          <description>Kill 7 opponents from behind in a row without being spotted, any level, any difficulty</description>
          <image>http://tiles.xbox.com/tiles/uN/FA/0jc8P2NhbC8yCxt0GltTWTBGL2FjaC8wLzFGAAAAAVBQUP1v0aQ=.jpg</image>
          <gamerScore>20</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='Demon'>
          <description>Complete any campaign level without dying once on Normal, Heroic, or Legendary</description>
          <image>http://tiles.xbox.com/tiles/mp/rc/0jc8P2NhbC81FQRrXVBRFzBGL2FjaC8wL0EAAAABUFBQ-fOagQ==.jpg</image>
          <gamerScore>25</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='Go Ape Shiv'>
          <description>Kill an enraged, berserk Brute by melee, any level, any difficulty</description>
          <image>http://tiles.xbox.com/tiles/RY/C-/1Dc8P2NhbC8wFQRrXVBRFzBGL2FjaC8wL0QAAAABUFBQ+5CAXg==.jpg</image>
          <gamerScore>20</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='Counterpoint'>
          <description>Kill the sword carrier</description>
          <image>http://tiles.xbox.com/tiles/xp/ul/0zc8P2NhbC9DFQRrXVBRFzBGL2FjaC8wLzcAAAABUFBQ-Iqb3Q==.jpg</image>
          <gamerScore>5</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='Carjacking'>
          <description>Steal an occupied vehicle from an opponent and take it over</description>
          <image>http://tiles.xbox.com/tiles/VW/NF/0jc8P2NhbC9CFQRrXVBRFzBGL2FjaC8wLzYAAAABUFBQ-WpjTg==.jpg</image>
          <gamerScore>5</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='Violent Cartographer'>
          <description>Play every available default map and stay in each game until it’s over</description>
          <image>http://tiles.xbox.com/tiles/tA/uJ/1Dc8P2NhbC9DCBt0GltTWTBGL2FjaC8wLzI3AAAAAVBQUPumC6g=.jpg</image>
          <gamerScore>25</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='Rainman'>
          <description>Play every available game variant, with at least 3 other players, to the end of each game</description>
          <image>http://tiles.xbox.com/tiles/QT/ri/1zc8P2NhbC81Cxt0GltTWTBGL2FjaC8wLzFBAAAAAVBQUPjNOl0=.jpg</image>
          <gamerScore>25</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='Double Kill'>
          <description>Kill 2 opponents within 4 seconds</description>
          <image>http://tiles.xbox.com/tiles/L5/P9/0Dc8P2NhbC82FQRrXVBRFzBGL2FjaC8wL0IAAAABUFBQ-9KTNA==.jpg</image>
          <gamerScore>20</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='Triple Kill'>
          <description>Kill 3 opponents within 4 seconds</description>
          <image>http://tiles.xbox.com/tiles/AQ/Ko/1jc8P2NhbC9ACBt0GltTWTBGL2FjaC8wLzI0AAAAAVBQUPmHAh0=.jpg</image>
          <gamerScore>20</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='Killtacular'>
          <description>Kill 4 opponents within 4 seconds</description>
          <image>http://tiles.xbox.com/tiles/wi/H4/0Dc8P2NhbC9GCxt0GltTWTBGL2FjaC8wLzEyAAAAAVBQUP-XId4=.jpg</image>
          <gamerScore>20</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='Skewer Stopper'>
          <description>Kill the sword carrier after they kill 5 or more times in a row with the sword</description>
          <image>http://tiles.xbox.com/tiles/Te/Ar/0Tc8P2NhbC9ECBt0GltTWTBGL2FjaC8wLzIwAAAAAVBQUP4E4FE=.jpg</image>
          <gamerScore>20</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='Vigilante'>
          <description>Stop a Killing Spree of another player</description>
          <image>http://tiles.xbox.com/tiles/J-/Np/1Tc8P2NhbC9CCBt0GltTWTBGL2FjaC8wLzI2AAAAAVBQUPpG8zs=.jpg</image>
          <gamerScore>10</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='Air Traffic Control'>
          <description>Blow up the Banshee in flight while manned, with grenades or the rocket launcher</description>
          <image>http://tiles.xbox.com/tiles/rI/jn/1zc8P2NhbC9FFQRrXVBRFzBGL2FjaC8wLzEAAAABUFBQ+MiItw==.jpg</image>
          <gamerScore>20</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='Decorated Soldier'>
          <description>Get awarded at least 8 different medals in one non-team game</description>
          <image>http://tiles.xbox.com/tiles/yI/+C/0Dc8P2NhbC9MFQRrXVBRFzBGL2FjaC8wLzgAAAABUFBQ-62P0w==.jpg</image>
          <gamerScore>10</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='Ninja'>
          <description>In a non-team game, kill 5 people by melee, from behind</description>
          <image>http://tiles.xbox.com/tiles/js/N7/1zc8P2NhbC9CCxt0GltTWTBGL2FjaC8wLzE2AAAAAVBQUPhUw5I=.jpg</image>
          <gamerScore>20</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='Hired Gun'>
          <description>Kill an opponent who has the Legend achievement</description>
          <image>http://tiles.xbox.com/tiles/5N/A5/0zc8P2NhbC9ECxt0GltTWTBGL2FjaC8wLzEwAAAAAVBQUPwW0Pg=.jpg</image>
          <gamerScore>20</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
      </achievements>
    </game>
  </games>
</xboxGamer>

As you can tell, the xml file for a user can be quite large. I've only got two games on Xbox Live currently, but some people may have upwards of 30+ games and each game can have 40+ achievements. In the xml data, there will be some redundancies, but when I import the data into my database, I will make sure to check for uniqueness before adding pre-existing data. The scraper has many phases. First, the raw data import phase where I select the data using jQuery selectors and put the data into javascript objects. The next phase is generating xml from the objects. The last phase is importing the data into my SQL database so that data may be properly related to each user. For example, if two people have the same game, their game lists will simply point to the same game and not make another entry in the database. The same goes for achievements. All unique achievements that exist for each game will be added to the database. For each achievement that a user completes for a game, the system will add an entry that relates the user to the game and achievement.

XBox Live Data

20. September 2011 14:32 by Cameron in Programming, Xbox Live  //  Tags: , , , , , , , , , , ,   //   Comments

While my gaming social networking site, IGA: International Gamers' Alliance, is still under beta, I have been looking at ways to provide a more rich experience for my users. Lately I've been working on a way to gather data from XBox Live so that I can provide content to my users on IGA. I used to have a way to gather data from a RESTful API, using the official XBox Live API, that Microsoft employee, Duncan Mckenzie, used to host on his website. However, his service is no longer available. While there is an official XBox Live API, access to this API is restricted to those who are in the XBox Community Developer Program. Acceptance into this the XBCDP is very limited at the moment and it seems that only well known companies with sponsors receive membership into the program. 

While it would be very nice to get official access to the XBox Live API, it may be a while until I can get into the program. My social networking site, IGA, is still in beta and has much to be done on the roadmap to completion. Currently I am the only developer for the project and I am also in school so development is slow. Maybe once IGA is closer to completion, Microsoft will be more eager to accept me into the program. In the meantime, I have a solution for gathering data from XBox Live.

There are a couple of places to get data from XBox Live. There is the publicly available user's gamercard and the user's protected XBox.com profile. Getting data from the public gamercard is very easy. One could write a parser in PHP, C#, or even jQuery to get the different values from the HTML elements on the page. Retrieving data from a user's XBox.com profile requires a little more skill and resources. You cannot simply use cURL to remotely login to XBox.com since it has anti-bot mechanisms in place to check against the browser agent, browser cookies, and many other aspects that can't easily be manipulated with cURL. There is a remedy to this problem however.

This past summer, I learned about a headless webkit browser called PhantomJS from some co-workers while working on a project at work. We needed something that could run without a GUI on a server that could manipulate the DOM of a webpage. PhantomJS gave us exactly what we needed. After working on the project at work, it occurred to me that I could use PhantomJS in addition to jQuery to manipulate the DOM and screen scrape data from XBox.com.

I'm currently working on scripts to pull data from users' profiles including the users' games, the achievements earned in each game, and more information not publicly available on users' gamercards. Please understand though that screen scraping should only be done on a last resort and it is taxing on both ends to make numerous requests per day. I will implement some sort of data caching that will pull new data on a schedule to limit bandwidth usage. I plan to release this code to my Git hosting when it is finished. 

Month List

Tag cloud