Newest Progress on my XBL data scraper

31. October 2011 17:09 by Cameron in Xbox Live  //  Tags: , , , , , , ,   //   Comments

It's been a while since I last posted, but I have made some great progress on my XBL data scraper. I made some additions to include CDATA tags in the descriptions of achievements as some sample data that I ran into contained ampersand (&) as well as other xml entities in the text and it was creating xml parser errors. Also, I had to add some code to replace ampersand (&) with the xml entity & in xml attributes as I was getting more parser errors with my sample data. This goes the same for other xml entities. Now that these issues have been resolved, I believe my data collector can pull anyone's profile information, given enough time. After testing the scraper with one of my friend's profiles, he has nearly 100 games with 900+ achievements, the initial import into my database took on the order of 7-10 minutes. To deal with manageable updates, I will have to scan a subset of the most recent games played as I don't want it to take as long with incremental updates. The question is how frequently I should do these updates. The incremental updates have not been implemented yet, but will be put into effect soon. I have setup a test site for processing user profiles and viewing collected sample data. I will not make this link public, but if you're interested, shoot me an email and I can share the link with you.

Progress on my Xbox Live data scraper

11. October 2011 13:34 by Cameron in PhantomJS, Xbox Live  //  Tags: , , , , , ,   //   Comments

After many hours working with PhantomJS, I have finally managed to get all available data from xbox.com for my Xbox Live profile using jQuery selectors. The data is outputted as xml and will be imported into my database for quick retrieval and record of historical data. Here is what the xml for my profile looks like:

<xboxGamer>
  <profile gamertag='pcmantinker'>
    <gamerScore>545</gamerScore>
    <avatarSmall>http://avatar.xboxlive.com/avatar/pcmantinker/avatarpic-s.png</avatarSmall>
    <avatarLarge>http://avatar.xboxlive.com/avatar/pcmantinker/avatarpic-l.png</avatarLarge>
    <avatarBody>http://avatar.xboxlive.com/avatar/pcmantinker/avatar-body.png</avatarBody>
    <location>Louisiana</location>
    <bio>Hey guys,

I'm a senior in Computer Science at LSU. I love to play video games, compose piano music, and work on programming projects during my spare time.</bio>
    <motto>Geaux Tigers!</motto>
  </profile>
  <games>
    <game name='DiRT 2'>
      <link>http://marketplace.xbox.com/en-US/Title/1129121824</link>
      <smallImage>http://tiles.xbox.com/consoleAssets/434D0820/en-US/smallboxart.jpg</smallImage>
      <largeImage>http://tiles.xbox.com/consoleAssets/434D0820/en-US/largeboxart.jpg</largeImage>
      <lastPlayed>1/29/2011</lastPlayed>
      <earnedGamerScore>320</earnedGamerScore>
      <totalGamerScore>1000</totalGamerScore>
      <earnedAchievements>21</earnedAchievements>
      <totalAchievements>47</totalAchievements>
      <percentComplete>44</percentComplete>
      <achievementsLink>http://live.xbox.com/en-US/GameCenter/Achievements?titleId=1129121824</achievementsLink>
      <achievements>
        <achievement name='Jet Setter'>
          <description>You've raced in every location. You'll need a new passport soon!</description>
          <image>http://tiles.xbox.com/tiles/Yz/y8/1oCLiGJhbC8wCxsDGyxTWTIwL2FjaC8wLzFEAAAAAOfn5-mTPH8=.jpg</image>
          <gamerScore>15</gamerScore>
          <achievedOn>1/29/2011</achievedOn>
        </achievement>
        <achievement name='All-Star'>
          <description>You're an All-Star driver now!</description>
          <image>http://tiles.xbox.com/tiles/c9/9G/0ICLiGJhbC9CCxsDGyxTWTIwL2FjaC8wLzE2AAAAAOfn5-9p328=.jpg</image>
          <gamerScore>25</gamerScore>
          <achievedOn>1/1/2011</achievedOn>
        </achievement>
        <achievement name="Aren't You Popular!">
          <description>You're pretty popular! You're true friends with four superstars.</description>
          <image>http://tiles.xbox.com/tiles/Pz/3F/14CLiGJhbC9GCxsDGyxTWTIwL2FjaC8wLzEyAAAAAOfn5-jqPSM=.jpg</image>
          <gamerScore>15</gamerScore>
          <achievedOn>1/1/2011</achievedOn>
        </achievement>
        <achievement name='Just Drive!'>
          <description>You drove 100 miles (161 kilometres) without unlocking any other achievement.</description>
          <image>http://tiles.xbox.com/tiles/sP/wW/1oCLiGJhbC9ECBsDGyxTWTIwL2FjaC8wLzIwAAAAAOfn5-k5-Kw=.jpg</image>
          <gamerScore>15</gamerScore>
          <achievedOn>1/1/2011</achievedOn>
        </achievement>
        <achievement name='Terminated!'>
          <description>That's gotta hurt! You took terminal damage at over 140mph (225km/h).</description>
          <image>http://tiles.xbox.com/tiles/Fh/f9/1YCLiGJhbC82FQQcXCdRFzIwL2FjaC8wL0IAAAAA5+fn+tIXDQ==.jpg</image>
          <gamerScore>15</gamerScore>
          <achievedOn>8/14/2010</achievedOn>
        </achievement>
        <achievement name='Big Bucks!'>
          <description>Money! You've earned those Big Bucks!</description>
          <image>http://tiles.xbox.com/tiles/Sh/aE/1ICLiGJhbC9AFQQcXCdRFzIwL2FjaC8wLzQAAAAA5+fn+6sWUQ==.jpg</image>
          <gamerScore>25</gamerScore>
          <achievedOn>8/14/2010</achievedOn>
        </achievement>
        <achievement name='X Games Asia Champ'>
          <description>Banzai! You've won X Games Asia.</description>
          <image>http://tiles.xbox.com/tiles/7P/1v/14CLiGJhbC8yCBsDGyxTWTIwL2FjaC8wLzJGAAAAAOfn5-hA-fA=.jpg</image>
          <gamerScore>25</gamerScore>
          <achievedOn>8/14/2010</achievedOn>
        </achievement>
        <achievement name='Gate Crashed!'>
          <description>Every gate! That's sick! You hit every gate in a Gatecrasher event. </description>
          <image>http://tiles.xbox.com/tiles/Gc/wE/1ICLiGJhbC9ECxsDGyxTWTIwL2FjaC8wLzEwAAAAAOfn5-srzAU=.jpg</image>
          <gamerScore>10</gamerScore>
          <achievedOn>8/14/2010</achievedOn>
        </achievement>
        <achievement name='Friends For Life'>
          <description>You became true friends with a superstar!</description>
          <image>http://tiles.xbox.com/tiles/ij/Tk/1YCLiGJhbC9FCxsDGyxTWTIwL2FjaC8wLzExAAAAAOfn5-rLNJY=.jpg</image>
          <gamerScore>10</gamerScore>
          <achievedOn>8/11/2010</achievedOn>
        </achievement>
        <achievement name='Block Party'>
          <description>You're a Blockbuster. You've beaten Ken in a Throwdown.</description>
          <image>http://tiles.xbox.com/tiles/mt/ce/04CLiGJhbC83CxsDGyxTWTIwL2FjaC8wLzFDAAAAAOfn5-wx14Y=.jpg</image>
          <gamerScore>15</gamerScore>
          <achievedOn>8/10/2010</achievedOn>
        </achievement>
        <achievement name='Piece Of Cake'>
          <description>One challenge down! You've won a Throwdown.</description>
          <image>http://tiles.xbox.com/tiles/4C/em/0YCLiGJhbC9DCxsDGyxTWTIwL2FjaC8wLzE3AAAAAOfn5-6JJ-w=.jpg</image>
          <gamerScore>10</gamerScore>
          <achievedOn>8/10/2010</achievedOn>
        </achievement>
        <achievement name='That All You Got?'>
          <description>You beat a Superstar's time in a Timed Throwdown.</description>
          <image>http://tiles.xbox.com/tiles/1P/RY/04CLiGJhbC9GCRsDGyxTWTIwL2FjaC8wLzMyAAAAAOfn5-x39Mg=.jpg</image>
          <gamerScore>15</gamerScore>
          <achievedOn>8/10/2010</achievedOn>
        </achievement>
        <achievement name='Head Over Wheels'>
          <description>That's some hard charging! You rolled your vehicle but still won the race.</description>
          <image>http://tiles.xbox.com/tiles/8Q/uC/1YCLiGJhbC9MFQQcXCdRFzIwL2FjaC8wLzgAAAAA5+fn+q0L6g==.jpg</image>
          <gamerScore>10</gamerScore>
          <achievedOn>8/10/2010</achievedOn>
        </achievement>
        <achievement name='Getting A Reputation'>
          <description>You're getting a reputation out there. You've reached Level 15.</description>
          <image>http://tiles.xbox.com/tiles/fc/th/04CLiGJhbC9NCxsDGyxTWTIwL2FjaC8wLzE5AAAAAOfn5-xOy2E=.jpg</image>
          <gamerScore>15</gamerScore>
          <achievedOn>8/9/2010</achievedOn>
        </achievement>
        <achievement name='Gut It Out!'>
          <description>It’s the result that counts! You won an event without winning a single race.</description>
          <image>http://tiles.xbox.com/tiles/-x/+l/1oCLiGJhbC9DFQQcXCdRFzIwL2FjaC8wLzcAAAAA5+fn+Yof5A==.jpg</image>
          <gamerScore>10</gamerScore>
          <achievedOn>8/9/2010</achievedOn>
        </achievement>
        <achievement name='Pro'>
          <description>You've proved yourself in the Pro division. Now take it to the next level.</description>
          <image>http://tiles.xbox.com/tiles/xt/Zn/0oCLiGJhbC9BCxsDGyxTWTIwL2FjaC8wLzE1AAAAAOfn5-1I1to=.jpg</image>
          <gamerScore>15</gamerScore>
          <achievedOn>8/9/2010</achievedOn>
        </achievement>
        <achievement name='Keep It Clean!'>
          <description>This game has damage? You won a race without crashing!</description>
          <image>http://tiles.xbox.com/tiles/ox/7c/14CLiGJhbC81FQQcXCdRFzIwL2FjaC8wL0EAAAAA5+fn+PMeuA==.jpg</image>
          <gamerScore>20</gamerScore>
          <achievedOn>8/6/2010</achievedOn>
        </achievement>
        <achievement name='X Games Europe Champ'>
          <description>Blimey! You've won X Games Europe.</description>
          <image>http://tiles.xbox.com/tiles/Wf/RO/1YCLiGJhbC8xCBsDGyxTWTIwL2FjaC8wLzJFAAAAAOfn5-ph9EU=.jpg</image>
          <gamerScore>20</gamerScore>
          <achievedOn>8/5/2010</achievedOn>
        </achievement>
        <achievement name='Rookie Coming Through!'>
          <description>You gotta start somewhere. You've raced in your first DiRT Tour race.</description>
          <image>http://tiles.xbox.com/tiles/VS/6H/04CLiGJhbC9ACxsDGyxTWTIwL2FjaC8wLzE0AAAAAOfn5-yoLkk=.jpg</image>
          <gamerScore>10</gamerScore>
          <achievedOn>6/25/2010</achievedOn>
        </achievement>
        <achievement name='Nailed an Event'>
          <description>You've won an event. Now it's time to take on the world.</description>
          <image>http://tiles.xbox.com/tiles/he/8d/1ICLiGJhbC83FQQcXCdRFzIwL2FjaC8wL0MAAAAA5+fn+zLvng==.jpg</image>
          <gamerScore>10</gamerScore>
          <achievedOn></achievedOn>
        </achievement>
        <achievement name='Gnarly!'>
          <description>Flashy finish! You crossed the finish line with no more than two wheels touching the ground!</description>
          <image>http://tiles.xbox.com/tiles/Yv/Ni/1ICLiGJhbC9NFQQcXCdRFzIwL2FjaC8wLzkAAAAA5+fn+03zeQ==.jpg</image>
          <gamerScore>15</gamerScore>
          <achievedOn></achievedOn>
        </achievement>
        <achievement name='Dedication'>
          <description>Win every event in the DiRT Tour.</description>
          <image>http://tiles.xbox.com/tiles/2b/eV/0jc8P2NhbC9BFQQcXCdRFzIwL2FjaC8wLzUAAAABUFBQ-bq3wg==.jpg</image>
          <gamerScore>80</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='The Full Set'>
          <description>Drive every vehicle in the game.</description>
          <image>http://tiles.xbox.com/tiles/bL/60/0Dc8P2NhbC9CFQQcXCdRFzIwL2FjaC8wLzYAAAABUFBQ-5u+dw==.jpg</image>
          <gamerScore>30</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name="You're The Best, Around!">
          <description>Win a race at every track in the game (in any game mode).</description>
          <image>http://tiles.xbox.com/tiles/fF/1O/1jc8P2NhbC8wFQQcXCdRFzIwL2FjaC8wL0QAAAABUFBQ+WFdZw==.jpg</image>
          <gamerScore>20</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='Dominated!'>
          <description>Win a domination event, getting the maximum available points.</description>
          <image>http://tiles.xbox.com/tiles/76/Wu/1zc8P2NhbC8xFQQcXCdRFzIwL2FjaC8wL0UAAAABUFBQ+IGl9A==.jpg</image>
          <gamerScore>15</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='On The Bubble'>
          <description>Be second to last at each elimination point but go on to win a Last Man Standing race.</description>
          <image>http://tiles.xbox.com/tiles/Wq/yP/1Tc8P2NhbC8yFQQcXCdRFzIwL2FjaC8wL0YAAAABUFBQ+qCsQQ==.jpg</image>
          <gamerScore>15</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='Best Friends Forever!'>
          <description>Become true friends with every superstar.</description>
          <image>http://tiles.xbox.com/tiles/rJ/zU/0Tc8P2NhbC9HCxsDGyxTWTIwL2FjaC8wLzEzAAAAAVBQUP77nLA=.jpg</image>
          <gamerScore>30</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='I like a Challenge'>
          <description>Win 12 Throwdowns.</description>
          <image>http://tiles.xbox.com/tiles/7m/pw/1Tc8P2NhbC9MCxsDGyxTWTIwL2FjaC8wLzE4AAAAAVBQUPpfavI=.jpg</image>
          <gamerScore>20</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='As Good As It Gets'>
          <description>Reach Level 30 in the DiRT Tour.</description>
          <image>http://tiles.xbox.com/tiles/vH/8u/1zc8P2NhbC81CxsDGyxTWTIwL2FjaC8wLzFBAAAAAVBQUPgBf6A=.jpg</image>
          <gamerScore>30</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='Whose Backyard?'>
          <description>Beat Travis Pastrana in a Throwdown.</description>
          <image>http://tiles.xbox.com/tiles/CX/YP/1Tc8P2NhbC82CxsDGyxTWTIwL2FjaC8wLzFCAAAAAVBQUPogdhU=.jpg</image>
          <gamerScore>15</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='World Tour Winner'>
          <description>Win all the World Tour events.</description>
          <image>http://tiles.xbox.com/tiles/8J/2t/0Dc8P2NhbC8xCxsDGyxTWTIwL2FjaC8wLzFFAAAAAVBQUP+Cnew=.jpg</image>
          <gamerScore>50</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='X Games Champ!'>
          <description>Win all the X Games events.</description>
          <image>http://tiles.xbox.com/tiles/RZ/SM/0jc8P2NhbC8yCxsDGyxTWTIwL2FjaC8wLzFGAAAAAVBQUP2jlFk=.jpg</image>
          <gamerScore>50</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='Secret Achievement'>
          <description>Continue playing to unlock this secret achievement.</description>
          <image>/Content/Images/HiddenAchievement.png</image>
          <gamerScore>75</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='Five Of Your Best'>
          <description>Win 5 Pro Tour races over Games for Windows - LIVE.</description>
          <image>http://tiles.xbox.com/tiles/ll/Qm/0jc8P2NhbC9GCBsDGyxTWTIwL2FjaC8wLzIyAAAAAVBQUP0JVIo=.jpg</image>
          <gamerScore>15</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='Tourney'>
          <description>Take part in four tournaments.</description>
          <image>http://tiles.xbox.com/tiles/Ba/zG/0zc8P2NhbC9HCBsDGyxTWTIwL2FjaC8wLzIzAAAAAVBQUPzprBk=.jpg</image>
          <gamerScore>20</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='Famous'>
          <description>Reach Online Fame Level 15.</description>
          <image>http://tiles.xbox.com/tiles/-E/dk/1jc8P2NhbC9ACBsDGyxTWTIwL2FjaC8wLzI0AAAAAVBQUPlLR+A=.jpg</image>
          <gamerScore>20</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='Notorious'>
          <description>Reach Online Fame Level 30.</description>
          <image>http://tiles.xbox.com/tiles/b7/+E/1zc8P2NhbC9BCBsDGyxTWTIwL2FjaC8wLzI1AAAAAVBQUPirv3M=.jpg</image>
          <gamerScore>40</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='Rush Hour'>
          <description>Win a Pro Tour race against 7 competitors over Games for Windows - LIVE.</description>
          <image>http://tiles.xbox.com/tiles/2r/al/1Tc8P2NhbC9CCBsDGyxTWTIwL2FjaC8wLzI2AAAAAVBQUPqKtsY=.jpg</image>
          <gamerScore>10</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='Mix It Up!'>
          <description>Play in every discipline over Games for Windows - LIVE (Pro Tour or Jam Session).</description>
          <image>http://tiles.xbox.com/tiles/SU/5F/1Dc8P2NhbC9DCBsDGyxTWTIwL2FjaC8wLzI3AAAAAVBQUPtqTlU=.jpg</image>
          <gamerScore>15</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='In His Footsteps'>
          <description>Win 25 Rally races in the DiRT Tour.</description>
          <image>http://tiles.xbox.com/tiles/R1/pi/1zc8P2NhbC9MCBsDGyxTWTIwL2FjaC8wLzI4AAAAAVBQUPhNWls=.jpg</image>
          <gamerScore>25</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='With Great Honor'>
          <description>Achieve a 'Cautious' impact rating in Multiplayer races (Pro Tour or Jam Session).</description>
          <image>http://tiles.xbox.com/tiles/1K/KC/1jc8P2NhbC9NCBsDGyxTWTIwL2FjaC8wLzI5AAAAAVBQUPmtosg=.jpg</image>
          <gamerScore>15</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='Mirra Image'>
          <description>Beat Dave Mirra in a Throwdown.</description>
          <image>http://tiles.xbox.com/tiles/FU/88/1Tc8P2NhbC81CBsDGyxTWTIwL2FjaC8wLzJBAAAAAVBQUPoTTwk=.jpg</image>
          <gamerScore>15</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name="I Ain't Afraid Of No Ghost!">
          <description>Choose a ghost, download and beat it!</description>
          <image>http://tiles.xbox.com/tiles/oE/Yd/1zc8P2NhbC82CBsDGyxTWTIwL2FjaC8wLzJCAAAAAVBQUPgyRrw=.jpg</image>
          <gamerScore>10</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='Dialled In'>
          <description>Win a race using a custom vehicle setup.</description>
          <image>http://tiles.xbox.com/tiles/M7/79/1jc8P2NhbC83CBsDGyxTWTIwL2FjaC8wLzJDAAAAAVBQUPnSvi8=.jpg</image>
          <gamerScore>5</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='Two Cups One Girl'>
          <description>Win two team events with the same female team-mate.</description>
          <image>http://tiles.xbox.com/tiles/yl/Vf/0zc8P2NhbC8wCBsDGyxTWTIwL2FjaC8wLzJEAAAAAVBQUPxwVdY=.jpg</image>
          <gamerScore>10</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='X Games America Champ'>
          <description>Win X Games America.</description>
          <image>http://tiles.xbox.com/tiles/8l/xo/1zc8P2NhbC9ECRsDGyxTWTIwL2FjaC8wLzMwAAAAAVBQUPhHXO4=.jpg</image>
          <gamerScore>30</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='Showboating'>
          <description>Bask in your success at X Games America.</description>
          <image>http://tiles.xbox.com/tiles/Ya/SI/1jc8P2NhbC9FCRsDGyxTWTIwL2FjaC8wLzMxAAAAAVBQUPmnpH0=.jpg</image>
          <gamerScore>20</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
      </achievements>
    </game>
    <game name='Halo 2 (PC)'>
      <link>http://marketplace.xbox.com/en-US/Title/1297287183</link>
      <smallImage>http://tiles.xbox.com/consoleAssets/4D53080F/en-US/smallboxart.jpg</smallImage>
      <largeImage>http://tiles.xbox.com/consoleAssets/4D53080F/en-US/largeboxart.jpg</largeImage>
      <lastPlayed>7/3/2010</lastPlayed>
      <earnedGamerScore>225</earnedGamerScore>
      <totalGamerScore>1000</totalGamerScore>
      <earnedAchievements>12</earnedAchievements>
      <totalAchievements>41</totalAchievements>
      <percentComplete>29</percentComplete>
      <achievementsLink>http://live.xbox.com/en-US/GameCenter/Achievements?titleId=1297287183</achievementsLink>
      <achievements>
        <achievement name='Cairo Station'>
          <description>Complete Cairo Station at Normal, Heroic, or Legendary</description>
          <image>http://tiles.xbox.com/tiles/4D/OV/14CLiGJhbC9BFQRrXVBRFzBGL2FjaC8wLzUAAAAA5+fn+Loz+w==.jpg</image>
          <gamerScore>30</gamerScore>
          <achievedOn>7/3/2010</achievedOn>
        </achievement>
        <achievement name='Assassin'>
          <description>Melee an opponent from behind</description>
          <image>http://tiles.xbox.com/tiles/ii/DX/04CLiGJhbC9HFQRrXVBRFzBGL2FjaC8wLzMAAAAA5+fn-PggkQ==.jpg</image>
          <gamerScore>10</gamerScore>
          <achievedOn>9/10/2008</achievedOn>
        </achievement>
        <achievement name='Sniper Kill'>
          <description>Kill an opponent with the sniper rifle or the beam rifle</description>
          <image>http://tiles.xbox.com/tiles/3k/E6/14CLiGJhbC9FCBt0GltTWTBGL2FjaC8wLzIxAAAAAOfn5-gVQcI=.jpg</image>
          <gamerScore>10</gamerScore>
          <achievedOn>9/10/2008</achievedOn>
        </achievement>
        <achievement name='Bonecracker'>
          <description>Hit and kill an opponent with a melee attack</description>
          <image>http://tiles.xbox.com/tiles/c8/t1/1oCLiGJhbC9AFQRrXVBRFzBGL2FjaC8wLzQAAAAA5+fn+VrLaA==.jpg</image>
          <gamerScore>10</gamerScore>
          <achievedOn>9/10/2008</achievedOn>
        </achievement>
        <achievement name='Running Riot'>
          <description>Kill 10 opponents in a row in the same game, without dying</description>
          <image>http://tiles.xbox.com/tiles/nn/lw/1oCLiGJhbC8wCxt0GltTWTBGL2FjaC8wLzFEAAAAAOfn5-lfeYI=.jpg</image>
          <gamerScore>20</gamerScore>
          <achievedOn>9/10/2008</achievedOn>
        </achievement>
        <achievement name='Killing Spree'>
          <description>Kill 5 opponents in a row in the same game, without dying</description>
          <image>http://tiles.xbox.com/tiles/d3/Eo/1YCLiGJhbC9FCxt0GltTWTBGL2FjaC8wLzExAAAAAOfn5-oHcWs=.jpg</image>
          <gamerScore>20</gamerScore>
          <achievedOn>9/10/2008</achievedOn>
        </achievement>
        <achievement name='The Arbiter'>
          <description>Complete The Arbiter at Normal, Heroic, or Legendary</description>
          <image>http://tiles.xbox.com/tiles/Gd/g3/0oCLiGJhbC9GFQRrXVBRFzBGL2FjaC8wLzIAAAAA5+fn-RjYAg==.jpg</image>
          <gamerScore>30</gamerScore>
          <achievedOn>9/5/2008</achievedOn>
        </achievement>
        <achievement name='Stick It'>
          <description>Stick an opponent with a plasma grenade and blow them up</description>
          <image>http://tiles.xbox.com/tiles/a0/gb/1YCLiGJhbC9GCBt0GltTWTBGL2FjaC8wLzIyAAAAAOfn5-o0SHc=.jpg</image>
          <gamerScore>5</gamerScore>
          <achievedOn>9/4/2008</achievedOn>
        </achievement>
        <achievement name='Roadkill'>
          <description>Run over, hit and kill an opponent with a vehicle</description>
          <image>http://tiles.xbox.com/tiles/Z5/LS/04CLiGJhbC83Cxt0GltTWTBGL2FjaC8wLzFDAAAAAOfn5-z9kns=.jpg</image>
          <gamerScore>10</gamerScore>
          <achievedOn>9/4/2008</achievedOn>
        </achievement>
        <achievement name='Flaming Ninja'>
          <description>Kill an opponent that has the Ninja achievement</description>
          <image>http://tiles.xbox.com/tiles/vD/Ls/1oCLiGJhbC83FQRrXVBRFzBGL2FjaC8wL0MAAAAA5+fn+cMypw==.jpg</image>
          <gamerScore>20</gamerScore>
          <achievedOn>9/3/2008</achievedOn>
        </achievement>
        <achievement name='Metropolis'>
          <description>Complete Metropolis at Normal, Heroic, or Legendary</description>
          <image>http://tiles.xbox.com/tiles/O5/Or/0oCLiGJhbC9BCxt0GltTWTBGL2FjaC8wLzE1AAAAAOfn5-2Ekyc=.jpg</image>
          <gamerScore>30</gamerScore>
          <achievedOn>9/3/2008</achievedOn>
        </achievement>
        <achievement name='Outskirts'>
          <description>Complete Outskirts at Normal, Heroic, or Legendary</description>
          <image>http://tiles.xbox.com/tiles/E3/ZN/0oCLiGJhbC9MCxt0GltTWTBGL2FjaC8wLzE4AAAAAOfn5-1idg8=.jpg</image>
          <gamerScore>30</gamerScore>
          <achievedOn>9/3/2008</achievedOn>
        </achievement>
        <achievement name='Oracle'>
          <description>Complete The Oracle at Normal, Heroic, or Legendary</description>
          <image>http://tiles.xbox.com/tiles/HT/ub/1jc8P2NhbC9DCxt0GltTWTBGL2FjaC8wLzE3AAAAAVBQUPm0OwE=.jpg</image>
          <gamerScore>30</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='Delta Halo'>
          <description>Complete Delta Halo at Normal, Heroic, or Legendary</description>
          <image>http://tiles.xbox.com/tiles/W3/di/0Tc8P2NhbC9NFQRrXVBRFzBGL2FjaC8wLzkAAAABUFBQ-k13QA==.jpg</image>
          <gamerScore>30</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='Regret'>
          <description>Complete Regret at Normal, Heroic, or Legendary</description>
          <image>http://tiles.xbox.com/tiles/9D/PD/1Tc8P2NhbC82Cxt0GltTWTBGL2FjaC8wLzFCAAAAAVBQUPrsM+g=.jpg</image>
          <gamerScore>30</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='Sacred Icon'>
          <description>Complete Sacred Icon at Normal, Heroic, or Legendary</description>
          <image>http://tiles.xbox.com/tiles/Dd/hh/0Dc8P2NhbC8xCxt0GltTWTBGL2FjaC8wLzFFAAAAAVBQUP9O2BE=.jpg</image>
          <gamerScore>30</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='Quarantine Zone'>
          <description>Complete Quarantine Zone at Normal, Heroic, or Legendary</description>
          <image>http://tiles.xbox.com/tiles/gN/dc/1Dc8P2NhbC9NCxt0GltTWTBGL2FjaC8wLzE5AAAAAVBQUPtz15w=.jpg</image>
          <gamerScore>30</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='Gravemind'>
          <description>Complete Gravemind at Normal, Heroic, or Legendary</description>
          <image>http://tiles.xbox.com/tiles/1n/hf/1Tc8P2NhbC8xFQRrXVBRFzBGL2FjaC8wL0UAAAABUFBQ+nB4zQ==.jpg</image>
          <gamerScore>30</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='Uprising'>
          <description>Complete Uprising at Normal, Heroic, or Legendary</description>
          <image>http://tiles.xbox.com/tiles/kv/pI/1zc8P2NhbC9BCBt0GltTWTBGL2FjaC8wLzI1AAAAAVBQUPhn+o4=.jpg</image>
          <gamerScore>30</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='High Charity'>
          <description>Complete High Charity at Normal, Heroic, or Legendary</description>
          <image>http://tiles.xbox.com/tiles/Ke/dO/1jc8P2NhbC9NCBt0GltTWTBGL2FjaC8wLzI5AAAAAVBQUPlh5zU=.jpg</image>
          <gamerScore>30</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='The Great Journey'>
          <description>Complete The Great Journey at Normal, Heroic, or Legendary</description>
          <image>http://tiles.xbox.com/tiles/+O/kK/0zc8P2NhbC9HCBt0GltTWTBGL2FjaC8wLzIzAAAAAVBQUPwl6eQ=.jpg</image>
          <gamerScore>30</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='Warrior'>
          <description>Complete the game on Normal difficulty</description>
          <image>http://tiles.xbox.com/tiles/uh/+u/1zc8P2NhbC9MCBt0GltTWTBGL2FjaC8wLzI4AAAAAVBQUPiBH6Y=.jpg</image>
          <gamerScore>100</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='Hero'>
          <description>Complete the game on Heroic difficulty</description>
          <image>http://tiles.xbox.com/tiles/Y3/F+/1zc8P2NhbC8yFQRrXVBRFzBGL2FjaC8wL0YAAAABUFBQ+FFxeA==.jpg</image>
          <gamerScore>50</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='Legend'>
          <description>Complete the game on Legendary difficulty</description>
          <image>http://tiles.xbox.com/tiles/qD/K6/1Dc8P2NhbC9ACxt0GltTWTBGL2FjaC8wLzE0AAAAAVBQUPuVMrQ=.jpg</image>
          <gamerScore>50</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='King of the Scarab'>
          <description>Acquire the Scarab gun on the Metropolis level, any difficulty</description>
          <image>http://tiles.xbox.com/tiles/Ud/kY/0Tc8P2NhbC9HCxt0GltTWTBGL2FjaC8wLzEzAAAAAVBQUP432U0=.jpg</image>
          <gamerScore>20</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='Silent but Deadly'>
          <description>Kill 7 opponents from behind in a row without being spotted, any level, any difficulty</description>
          <image>http://tiles.xbox.com/tiles/uN/FA/0jc8P2NhbC8yCxt0GltTWTBGL2FjaC8wLzFGAAAAAVBQUP1v0aQ=.jpg</image>
          <gamerScore>20</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='Demon'>
          <description>Complete any campaign level without dying once on Normal, Heroic, or Legendary</description>
          <image>http://tiles.xbox.com/tiles/mp/rc/0jc8P2NhbC81FQRrXVBRFzBGL2FjaC8wL0EAAAABUFBQ-fOagQ==.jpg</image>
          <gamerScore>25</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='Go Ape Shiv'>
          <description>Kill an enraged, berserk Brute by melee, any level, any difficulty</description>
          <image>http://tiles.xbox.com/tiles/RY/C-/1Dc8P2NhbC8wFQRrXVBRFzBGL2FjaC8wL0QAAAABUFBQ+5CAXg==.jpg</image>
          <gamerScore>20</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='Counterpoint'>
          <description>Kill the sword carrier</description>
          <image>http://tiles.xbox.com/tiles/xp/ul/0zc8P2NhbC9DFQRrXVBRFzBGL2FjaC8wLzcAAAABUFBQ-Iqb3Q==.jpg</image>
          <gamerScore>5</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='Carjacking'>
          <description>Steal an occupied vehicle from an opponent and take it over</description>
          <image>http://tiles.xbox.com/tiles/VW/NF/0jc8P2NhbC9CFQRrXVBRFzBGL2FjaC8wLzYAAAABUFBQ-WpjTg==.jpg</image>
          <gamerScore>5</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='Violent Cartographer'>
          <description>Play every available default map and stay in each game until it’s over</description>
          <image>http://tiles.xbox.com/tiles/tA/uJ/1Dc8P2NhbC9DCBt0GltTWTBGL2FjaC8wLzI3AAAAAVBQUPumC6g=.jpg</image>
          <gamerScore>25</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='Rainman'>
          <description>Play every available game variant, with at least 3 other players, to the end of each game</description>
          <image>http://tiles.xbox.com/tiles/QT/ri/1zc8P2NhbC81Cxt0GltTWTBGL2FjaC8wLzFBAAAAAVBQUPjNOl0=.jpg</image>
          <gamerScore>25</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='Double Kill'>
          <description>Kill 2 opponents within 4 seconds</description>
          <image>http://tiles.xbox.com/tiles/L5/P9/0Dc8P2NhbC82FQRrXVBRFzBGL2FjaC8wL0IAAAABUFBQ-9KTNA==.jpg</image>
          <gamerScore>20</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='Triple Kill'>
          <description>Kill 3 opponents within 4 seconds</description>
          <image>http://tiles.xbox.com/tiles/AQ/Ko/1jc8P2NhbC9ACBt0GltTWTBGL2FjaC8wLzI0AAAAAVBQUPmHAh0=.jpg</image>
          <gamerScore>20</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='Killtacular'>
          <description>Kill 4 opponents within 4 seconds</description>
          <image>http://tiles.xbox.com/tiles/wi/H4/0Dc8P2NhbC9GCxt0GltTWTBGL2FjaC8wLzEyAAAAAVBQUP-XId4=.jpg</image>
          <gamerScore>20</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='Skewer Stopper'>
          <description>Kill the sword carrier after they kill 5 or more times in a row with the sword</description>
          <image>http://tiles.xbox.com/tiles/Te/Ar/0Tc8P2NhbC9ECBt0GltTWTBGL2FjaC8wLzIwAAAAAVBQUP4E4FE=.jpg</image>
          <gamerScore>20</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='Vigilante'>
          <description>Stop a Killing Spree of another player</description>
          <image>http://tiles.xbox.com/tiles/J-/Np/1Tc8P2NhbC9CCBt0GltTWTBGL2FjaC8wLzI2AAAAAVBQUPpG8zs=.jpg</image>
          <gamerScore>10</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='Air Traffic Control'>
          <description>Blow up the Banshee in flight while manned, with grenades or the rocket launcher</description>
          <image>http://tiles.xbox.com/tiles/rI/jn/1zc8P2NhbC9FFQRrXVBRFzBGL2FjaC8wLzEAAAABUFBQ+MiItw==.jpg</image>
          <gamerScore>20</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='Decorated Soldier'>
          <description>Get awarded at least 8 different medals in one non-team game</description>
          <image>http://tiles.xbox.com/tiles/yI/+C/0Dc8P2NhbC9MFQRrXVBRFzBGL2FjaC8wLzgAAAABUFBQ-62P0w==.jpg</image>
          <gamerScore>10</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='Ninja'>
          <description>In a non-team game, kill 5 people by melee, from behind</description>
          <image>http://tiles.xbox.com/tiles/js/N7/1zc8P2NhbC9CCxt0GltTWTBGL2FjaC8wLzE2AAAAAVBQUPhUw5I=.jpg</image>
          <gamerScore>20</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='Hired Gun'>
          <description>Kill an opponent who has the Legend achievement</description>
          <image>http://tiles.xbox.com/tiles/5N/A5/0zc8P2NhbC9ECxt0GltTWTBGL2FjaC8wLzEwAAAAAVBQUPwW0Pg=.jpg</image>
          <gamerScore>20</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
      </achievements>
    </game>
  </games>
</xboxGamer>

As you can tell, the xml file for a user can be quite large. I've only got two games on Xbox Live currently, but some people may have upwards of 30+ games and each game can have 40+ achievements. In the xml data, there will be some redundancies, but when I import the data into my database, I will make sure to check for uniqueness before adding pre-existing data. The scraper has many phases. First, the raw data import phase where I select the data using jQuery selectors and put the data into javascript objects. The next phase is generating xml from the objects. The last phase is importing the data into my SQL database so that data may be properly related to each user. For example, if two people have the same game, their game lists will simply point to the same game and not make another entry in the database. The same goes for achievements. All unique achievements that exist for each game will be added to the database. For each achievement that a user completes for a game, the system will add an entry that relates the user to the game and achievement.

Passing information to and from webpages in PhantomJS

27. September 2011 11:01 by Cameron in javascript, PhantomJS  //  Tags: , , , , ,   //   Comments

Recently, I needed a way to pass dynamic content to and from webpages using PhantomJS as part of writing my screen scraper. I need the scraper to follow dynamic sets of links and scrape the data from each page. Since a webpage's scope is currently sand boxed, I had to find a way to pass data to and from webpages. With the addition of the new filesystem module in PhantomJS 1.3, it is now possible to pass data from the main scope to an individual page's scope. Any data that you want passed to a particular page should be saved as a javascript string to a javascript file. Then you can inject the javascript into the page on page.onLoadFinished so that the data is then accessible within the page's scope. For example:

var page = require('webpage').create(), 
     fs = require('fs'), 
     data = "var dataObject = { item: 'value' };", 
     fullpath;

fullpath = fs.workingDirectory + fs.separator + 'data.js';
// open file for writing
var dataFile = fs.open(fullpath, 'w');
dataFile.write(data);
dataFile.close();

// check that the file was successfully written
if(fs.size(fullpath) > 0) {
	console.log('File wrote successfully!');
	page.open('http://somesite.org/page.html');
	// put page data in a local variable
	var output = page.evaluate(function () {
		// print the output of the data object
		console.log(dataObject.item);
		return dataObject.item;
	});
	// output should be the same value as the page's dataObject.item
	console.log(output);
}
else {
	console.log('Error in writing the file!');
	phantom.exit();
}

page.onLoadFinished = function() {
	// inject the javascript data that we created earlier
	page.injectJS(fullpath);
}

For more information about PhantomJS' File System module, please visit: http://code.google.com/p/phantomjs/wiki/Interface#Filesystem_Module

While this solution may not be the best long term solution, it does provide a way to get data to and from your pages until official support for passing data to a webpage object becomes available in PhantomJS.

Installation of Postive SSL wildcard certificate

26. September 2011 01:27 by Cameron in Security, Web  //  Tags: , , , , , , , , ,   //   Comments

The other day, I splurged on getting a wildcard SSL certificate for my website, www.iga-home.net. I felt that it was important to secure content on my site for my users as sensitive data is sent when users login or post content to the site. I bought a wildcard SSL certificate since I wanted to be able to secure all subdomains of iga-home.net and not be restricted to just iga-home.net or www.iga-home.net. 

After I created the request on IIS for the certificate, I copied the output into my web browser on Comodo's website for requesting an SSL certificate. It seemed fairly straight forward. About 10-15 minutes later, I received an email with my SSL certificate attached in a zip file. I opened the zip file and saw my certificate with a .cert extension. I fiddled around for a while with trying to get this certificate installed through mmc and IIS's manager. When I tried to install my certificate through IIS, I kept receiving errors that IIS couldn't find my certificate request. I followed many tutorials and couldn't find a solution. I later found this page that said I should install these certificates first through mmc before I could install my purchased SSL certificate in IIS. After I installed these mentioned certificates, IIS accepted my certificate and I proceeded to adding SSL to my website. It's a shame that these certificates weren't bundled with the original zip file. It would have made life a lot easier.

My next task is to add a rewrite rule for sending all http requests to https requests. I also want to write a resource handler that caches remote resources on my server so that all resources are secure. In Google Chrome it will notify the user if some content displayed on a page is insecure and I want to remedy this problem. 

Take Screenshot of all HTML documents in a folder using PhantomJS

26. September 2011 01:14 by Cameron in javascript, PhantomJS, Programming  //  Tags: , , , ,   //   Comments

Recently I came across a question on stackoverflow that asked about how to take screenshots of all HTML files in a local folder. I have been playing with PhantomJS quite a bit lately and felt comfortable answering the question. Here is the code for those interested:

var page = require('webpage').create(), loadInProgress = false, fs = require('fs');
var htmlFiles = new Array();
console.log('working directory: ' + fs.workingDirectory);
var curdir = fs.list(fs.workingDirectory);

// loop through files and folders
for(var i = 0; i< curdir.length; i++)
{
	var fullpath = fs.workingDirectory + fs.separator + curdir[i];
	// check if item is a file
	if(fs.isFile(fullpath))
	{
		if(fullpath.indexOf('.html') != -1)
		{
			// show full path of file
			console.log('File path: ' + fullpath);
			htmlFiles.push(fullpath);
		}
	}
}

console.log('Number of Html Files: ' + htmlFiles.length);

// output pages as PNG
var pageindex = 0;

var interval = setInterval(function() {
	if (!loadInProgress && pageindex < htmlFiles.length) {
		console.log("image " + (pageindex + 1));
		page.open(htmlFiles[pageindex]);
	}
	if (pageindex == htmlFiles.length) {
		console.log("image render complete!");
		phantom.exit();
	}
}, 250);

page.onLoadStarted = function() {
	loadInProgress = true;
	console.log('page ' + (pageindex + 1) + ' load started');
};

page.onLoadFinished = function() {
	loadInProgress = false;
	page.render("images/output" + (pageindex + 1) + ".png");
	console.log('page ' + (pageindex + 1) + ' load finished');
	pageindex++;
}

The process is quite simple. First, I loop through all objects in the current working directory and check to see if each item is a file and whether it has the .html extension. Then I add each html file's filepath to an array that I later loop through to take the screenshots. A screenshot must be taken after the page is fully loaded so that the screenshot will contain actual content and not a blank image. This is done by saving the image on the page.onLoadFinished callback. The application loop for taking the screenshots inserts small 250ms delays between each request so that pages may fully load into the headless browser before advancing to the next page.

XBox Live Data

20. September 2011 14:32 by Cameron in Programming, Xbox Live  //  Tags: , , , , , , , , , , ,   //   Comments

While my gaming social networking site, IGA: International Gamers' Alliance, is still under beta, I have been looking at ways to provide a more rich experience for my users. Lately I've been working on a way to gather data from XBox Live so that I can provide content to my users on IGA. I used to have a way to gather data from a RESTful API, using the official XBox Live API, that Microsoft employee, Duncan Mckenzie, used to host on his website. However, his service is no longer available. While there is an official XBox Live API, access to this API is restricted to those who are in the XBox Community Developer Program. Acceptance into this the XBCDP is very limited at the moment and it seems that only well known companies with sponsors receive membership into the program. 

While it would be very nice to get official access to the XBox Live API, it may be a while until I can get into the program. My social networking site, IGA, is still in beta and has much to be done on the roadmap to completion. Currently I am the only developer for the project and I am also in school so development is slow. Maybe once IGA is closer to completion, Microsoft will be more eager to accept me into the program. In the meantime, I have a solution for gathering data from XBox Live.

There are a couple of places to get data from XBox Live. There is the publicly available user's gamercard and the user's protected XBox.com profile. Getting data from the public gamercard is very easy. One could write a parser in PHP, C#, or even jQuery to get the different values from the HTML elements on the page. Retrieving data from a user's XBox.com profile requires a little more skill and resources. You cannot simply use cURL to remotely login to XBox.com since it has anti-bot mechanisms in place to check against the browser agent, browser cookies, and many other aspects that can't easily be manipulated with cURL. There is a remedy to this problem however.

This past summer, I learned about a headless webkit browser called PhantomJS from some co-workers while working on a project at work. We needed something that could run without a GUI on a server that could manipulate the DOM of a webpage. PhantomJS gave us exactly what we needed. After working on the project at work, it occurred to me that I could use PhantomJS in addition to jQuery to manipulate the DOM and screen scrape data from XBox.com.

I'm currently working on scripts to pull data from users' profiles including the users' games, the achievements earned in each game, and more information not publicly available on users' gamercards. Please understand though that screen scraping should only be done on a last resort and it is taxing on both ends to make numerous requests per day. I will implement some sort of data caching that will pull new data on a schedule to limit bandwidth usage. I plan to release this code to my Git hosting when it is finished. 

Thoughts on Windows 8 Developer Preview

20. September 2011 13:30 by Cameron in Windows 8  //  Tags: , , , , , , , , , , , , , , ,   //   Comments

Last week I downloaded the Windows 8 Developer Preview in both the x64 and x86 editions. I first installed the x64 edition with the developer tools in a virtual machine in Oracle VirtualBox to get a feel for the operating system. After I had installed the DP, I immediately wanted to try out the new Metro UI applications. However, although the tiles were responding to my mouse clicks, they were not opening up in the virtual machine. I did some research to find out what might be the cause of this and learned that Metro UI apps require at least a 1024x768 screen resolution to run. I changed my screen resolution in my virtual machine to 1024x768 and voila, the Metro UI apps worked.

After getting a sense for the new Metro UI, I ventured into installing the 32 bit Windows 8 DP on my ASUS EEEPC T101MT Intel Atom netbook. The installation went smoothly as expected and I was brought into the Metro UI on login. My netbook has a 10.1 inch screen with a standard 1024x600 resolution so I had to apply a registry hack to get support for a 1024x768 resolution. After applying the registry hack and loading the EEEPC resolution changer, I was able to get my netbook running with a 1024x768 resolution. The higher resolution requires down scaling due to the smaller screen size, making things appear squished using the legacy UI. However, the Metro UI looks fairly nice with the higher resolution and most applications work well. 

I'd say overall, I'm pretty pleased with the first Windows 8 Developer Preview public release. I'm looking forward to seeing if Microsoft will actually add support for my smaller screen resolution in the upcoming releases. It will be great to see new features unfold as updates for the new operating system arrive.

Hackintosh Computer Build

23. August 2011 16:27 by Cameron in Hackintosh  //  Tags:   //   Comments

Last summer I built my first gaming rig with a somewhat substantial budget of about $1400. This was not my first computer build, but it was the first build that I had spent my own money on. My goal was to build a powerful machine that was hardware compatible with Mac OS X Snow Leopard and later. Initially I had some issues with my graphics card as the Fermi line wasn't supported by Apple until about halfway into Snow Leopard's life. In addition to OS X, I also boot Windows 7 x64 enterprise and Ubuntu 64bit 11.04, Natty Narwhal. As a developer, I enjoy using different operating systems to code for various projects.

Here's my original build from last summer:
Mac OS X version 10.6 Snow Leopard
http://www.amazon.com/Mac-OS-Snow-Leopard-10-6/dp/B002KG02QO/ref=dp_cp_ob_sw_title_3
Intel Core i7 875K 2.93GHz Quad Core CPU
http://www.newegg.com/Product/Product.aspx?Item=N82E16819116368
MSI P55 GD65 Motherboard
http://www.newegg.com/Product/Product.aspx?Item=N82E16813130239
ADATA XPG Gaming Series 4GB (2 x 2GB) 240-Pin DDR3 SDRAM DDR3 1600 (x 2)
http://www.newegg.com/Product/Product.aspx?Item=N82E16820211409
EVGA NVIDIA GTX 470 1280MB GDDR5
http://www.amazon.com/nVidia-GeForce-1280-PCI-Express-Video/dp/B003EM68MK (I originally bought this from Newegg. It has been discontinued there however)
1TB Samsung Spinpoint F3
http://www.newegg.com/Product/Product.aspx?Item=N82E16822152185
Seagate Momentus 5400.6 ST9500325AS 500GB 5400 RPM 8MB Cache 2.5" SATA 3.0Gb/s Internal Notebook Hard Drive -Bare Drive
http://www.newegg.com/Product/Product.aspx?Item=N82E16822148371
80GB Excelstore  (already owned)

60GB Hitachi (already owned)

650W Thermaltake PSU
http://www.newegg.com/Product/Product.aspx?Item=N82E16817153116
Samsung Blu-Ray Combo

http://www.newegg.com/Product/Product.aspx?Item=N82E16827151199 (this drive has been discontinued on newegg)
Cooler Master Hyper 212 Plus CPU Cooler
http://www.newegg.com/Product/Product.aspx?Item=N82E16835103065
Antec 200 Mid Tower case
http://www.newegg.com/Product/Product.aspx?Item=N82E16811129074

This summer, I made a few upgrades:
G.SKILL 16GB DDR3 1333 RAM
http://www.newegg.com/Product/Product.aspx?Item=N82E16820231417
850W Thermaltake PSU
Bought this in CompUSA (Tiger Direct) store
2 x OCZ 60GB SSD
http://www.newegg.com/Product/Product.aspx?Item=N82E16820227542
Mac OS X Lion
Mac App Store

Installing Snow Leopard
I used iBoot supported (10.3 kernel) to boot the retail Snow Leopard 10.6.0 installer and Multibeast to install the DSDT. I installed the ALC889 kext from Multibeast and the legacy AppleHDA to get audio. I had to use my 9500 GT to install SL because I was getting kernel panics with my GTX 470 without the proper enabler installed. After SL installed, I installed the modified Fermi Chameleon bootloader to support my GTX 470. I installed the JMicron 36xxx kext from kexts.com to get PATA support.

Installing Lion

Once Lion was released, I bought Lion from the Mac App Store and I used tonymacx86's xMove to create a USB installer for installing Lion. I did a clean install of Lion on one of my OCZ SSDs since I no longer needed my Snow Leopard installation, I wanted to start fresh and I wanted to have the performance boost of my new SSD. Also, my GTX 470 works without a hitch in Lion as Apple has been supporting Fermi cards since about Snow Leopard 10.6.4.

Here are some older photos of my computer when I first built the system. I'll upload some newer photos soon.

Gitting started with Git

15. August 2011 00:25 by Cameron in Git  //  Tags: , , , , , , , , ,   //   Comments

Git, created by Linus Torvalds, is a very high quality version control system. It was created with the task to manage the source tree of the Linux kernel. Torvalds didn't believe that pre-existing version control systems could give justice to the Linux kernel's source code given its massive size and collaborators so Torvalds created Git. If you are using other version control systems for your projects, consider reading this: http://whygitisbetterthanx.com/

This website explains the advantages in full of why Git is better than other version control systems available. 

Git is free and open source and is available for all platforms: Linux, Mac, Windows, Solaris, you name it

First, be sure to install git for your platform and then you can start playing around with different commands. Once you've installed git, here are a few references to get you started:  

http://book.git-scm.com/  

http://www.kernel.org/pub/software/scm/git/docs/  

Setting Up Git

In order to setup your environment for using a remote git repository, be sure to run these commands:

$ ssh-keygen -t rsa -C "youremail@site.com"

This command creates a public/private key pair for SSH. SSH is used by git to encrypt the connection to remote servers. When asked to where to save your public key, press enter. Then, when asked for a passphrase, leave it empty. Your screen should look like this:

Generating public/private rsa key pair.

Enter file in which to save the key (/home/cameron/.ssh/id_rsa): 

Created directory '/home/cameron/.ssh'.

Enter passphrase (empty for no passphrase): 

Enter same passphrase again: 

Your identification has been saved in /home/cameron/.ssh/id_rsa.

Your public key has been saved in /home/cameron/.ssh/id_rsa.pub.

After your public/private key have been setup, add your global user information:

$ git config --global user.name "Firstname Lastname"

$ git config --global user.email "your_email@youremail.com"

Now you are ready to clone a repository. If you run:

$ git clone git@git.tinksoft.net:test.git

A new directory will be created for the git repository, test, and all of the remote files in the repository will be downloaded into that directory.

Git Command Basics

A few common commands to git are cloning repositories, committing to repositories, pushing to repositories, and pulling from repositories. If you've worked with subversion before, "git clone" is like subversion checkout. It literally clones the remote repository in its current state to your local repository. However, "git commit" is not like subversion commit. When you commit to a git repository, you are only committing to your local repository until you push to the remote repository. Using "git push" is like subversion commit and will push your changes to the remote repository. On the first push, you need to run the command "git push origin <branch name>". This tells git to push the origin to the branch that you specify. After that first push, you can run "git push" thereafter. If you choose to switch branches later on, you simply need to run the original command and specify your origin branch. Similarly, "git pull" behaves like subversion update and pulls down changes from your remote repository into your locally cloned repository. The same  applies to the first "git pull" as does the first "git push". Git needs to know which branch to pull from.  

One thing about pushing and pulling is that if you are working in a team and multiple people are pushing and pulling to the remote repository, you may be required to pull before you push out your changes. Don't worry though. If you have a conflict with your changes, your code will not be overwritten. Git has a conflict resolution tool where you can choose which changes to accept. Another thing that is good practice is to always run "git status" before committing and pushing to your repository. This will allow you to confirm that you are indeed committing files that should be committed to your repository. Also, whatever shows up in a commit log will be pushed to your remote repository when you push our your changes. Be sure to only push out working code and not break the build for your team.

A few advanced commands include "git branch <branch name>" (branches the repository at its current state), "git merge -s ours <branch>" (merges a branch with current branch), and "git checkout <branch name>" (changes current working branch). Please be sure to read up on these commands so that you know how to use them correctly. In a project repository, you don't want to create unnecessary branches, merge branches incorrectly, or lose changes when switching branches. Another advanced topic is to create a .gitignore file for your repository and put all files that git should ignore into this file. Each file should be on a separate line. This can be helpful if you don't want files such as database configurations to be pushed to your remote repository. 

For more information about git, be sure to read the references I listed above and also check out some books on git for a more in depth discussion.

Automatic builds in Jenkins from Git

11. August 2011 03:05 by Cameron in Continuous Integration, Git, Indefero, Jenkins  //  Tags: , , , , ,   //   Comments

Today I discovered how to run automated builds from Git post-receive hooks. Git has different hooks that you can trigger at various stages in the commit/push cycle. A full list of git hooks can be found here: http://www.kernel.org/pub/software/scm/git/docs/githooks.html

I found a very nice ruby script that does the trick of triggering an automatic build in Jenkins here: http://lostechies.com/jasonmeridth/2009/03/24/adding-a-git-post-receive-hook-to-fire-off-hudson-ci-server/

Here is the script:

#!/usr/bin/env ruby
#
while (input = STDIN.read) != ''
   rev_old, rev_new, ref = input.split(" ")
   if ref == "refs/heads/master"

       url="http://yourhudsondomain.com/job/job_name_here/build?delay=0sec"

       puts "Run Hudson build for job_name_here application"
       `wget #{url} > /dev/null 2>&1`
   end
end

I'm sure you could write a bash script to do the same thing if you wanted to, but the original author preferred to use Ruby.

I'm glad that automatic builds finally work. I struggled for quite some time on this issue. I was looking in the wrong place. The web interface that I use for git, Indefero, has a place for post-commit hook web urls, but the problem was that post-commit hooks don't behave the same in git as they do in subversion. I didn't want to trigger builds on post-commit in git but rather when someone pushes their commits to the server. If you have scm polling enabled for your job, you no longer need this after you've configured post-receive hooks.

Month List

Tag cloud