Knockout Observable Array Performance Improvments

Today, I was working on a project that makes AJAX calls to an API and noticed that the application was very slow and unresponsive during the AJAX calls and ko updates. I checked that the requests weren't taking too long in Chrome's developer tools and found they were at max taking 300ms and minimum of 20ms to return. This was very odd behavior since the AJAX requests weren't blocking.

After some research, I found that when updating ko.observableArrays, you don't want to push each individual item to the array if you can help it. Each time an item is pushed to a ko.observableArray, it notifies its subscribers and depending on how many DOM elements you have on your page, that can be rather taxing. I had code like this:

$.ajax({
	url: 'api/Document/',
	method: 'GET',
	dataType: 'json',
	success: function(data) {
		Documents([])
		for(var i = 0; i < data.length; i++)
			Documents.push(data[i]);
	}
});

To fix this, you need to create temporary arrays to push your items to and then assign the ko.observableArray to the contents of the temporary array. Your code should look more like this:

$.ajax({
	url: 'api/Document/',
	method: 'GET',
	dataType: 'json',
	success: function(data) {
		var documents = [];
		for(var i = 0; i < data.length; i++)
			documents.push(data[i]);	

		Documents(documents);
	}
});

Convert JavaScript Escaped Characters to HTML/XML Entities

11. January 2012 01:44 by Cameron in javascript  //  Tags: , , , , , ,   //   Comments

Recently, I needed a way to convert JavaScript escaped characters to HTML/XML entities for saving files in UTF-8 encoding without saving the special characters to disk. Luckily this was pretty easy. I used JavaScript's built-in function escape() to escape the input text and used regular expressions to find each escaped value and replace them in the input text with their new HTML/XML entity equivalent. For example, the text "Copyright Some Company ©" would escape to "Copyright%20Some%20Company%20%A9" using JavaScript's built-in function escape(). After converting the escaped characters to entities, the result would be "Copyright&#x20;Some&#x20;Company&#x20;&#xA9;". When saved in a database and rendered as HTML to a webpage, it would display the original message "Copyright Some Company ©". Here's the code for anyone interested:

// convert escaped characters to html/xml entities
function normalizeText(text)
{
	// escape text for special characters
	var esc = escape(text);
	var simpleCharsRegex = /%[0-9A-Fa-f][0-9A-Fa-f]/g;
	var specialCharsRegex = /%u[0-9A-Fa-f][0-9A-Fa-f][0-9A-Fa-f][0-9A-Fa-f]/g;
	var simpleCharsMatch = esc.match(simpleCharsRegex);
	var specialCharsMatch = esc.match(specialCharsRegex);
	if(simpleCharsMatch!=null)
	{
		for(var i = 0; i < simpleCharsMatch.length; i++)
		{
			var temp = simpleCharsMatch[i];
			simpleCharsMatch[i] = simpleCharsMatch[i].replace(/%/, 'x') + ';';
			esc = esc.replace(temp, simpleCharsMatch[i]);
		}
	}
	if(specialCharsMatch!=null)
	{
		for(var i = 0; i < specialCharsMatch.length; i++)
		{
			var temp = specialCharsMatch[i];
			specialCharsMatch[i] = specialCharsMatch[i].replace(/%u/, 'x') + ';';
			esc = esc.replace(temp, specialCharsMatch[i]);
		}
	}
	return esc;
}

I understand there may be a more efficient way to do this, but this code does work in a pinch. Please let me know in the comments if you have any suggestions for improving this code.

Nearly finished with my XBL Scraper and starting my PSN Scraper

3. December 2011 14:31 by Cameron in PhantomJS, Xbox Live, PSN  //  Tags: , , , , , , , , , ,   //   Comments

I've made a ton of progress on my XBL scraper over the past couple of months and am nearing completion. Features include grabbing a user's full list of games and achievements for all time as well as pulling a subset of information based on a date range. I implemented the date range feature because I wanted to speed up incremental updates. One of my beta testers has reported it takes about 15-20 minutes to do a full dump of his information. He also has over 100 games with about 40-60 achievements each.

Right now, my XBL scraper only pulls down information for the currently logged in user. I had thought about having my users on my site provide their login information for pulling down their achievements and recently played games, but there are huge security concerns with me storing Xbox LIVE user names and passwords in my database. One disadvantage to me pulling information from other user profiles is that any achievement that I have not unlocked the image will be returned as a grayscale image. This is not a huge issue though. I'm more leaning towards using my own account to get user data instead of other users' accounts because I don't want to be burdened with security of storing their credentials on my server. If people want color images for achievements, I will just need the community to submit their color images for their achievements to the database. I have also been thinking of a way to have my script run idle on my server while waiting for input from the caller. This will hopefully speed up processing of the data.

On the PSN front, I have begun analyzing various data sources for getting trophy data and recently played games for a PSNID. I've started developing a similar script for pulling information for a PSNID. The PSN scraper is still in its infancy, but it is making steady progress. With my analysis of PSN trophy images, it appears that images for trophies will be returned as color images for public trophy profiles and personal trophy profiles. However, the locked trophies show a locked image next to the trophy. I'm hoping to get community involvement to submit complete trophy images for the locked images. With the way that PSN shows earned trophies, with enough people in the community, the missing images should easily be filled in automatically.

I plan to release premium subscritpion APIs for these scripts in the future. Please stay tuned for more information.

Simple Options Parser for PhantomJS

18. November 2011 14:33 by Cameron in javascript, PhantomJS, Programming  //  Tags: , , ,   //   Comments

Recently I needed a way to parse command line options with PhantomJS. I didn't see anything else on the web that allowed for abitrary ordering of command line arguments to PhantomJS scripts so I made my own. Here's the code for those interested:

// argument results
var a1, a2, a3, a4;

function optionParser() {	
	var opt = 0;
	while((opt < phantom.args.length) && (phantom.args[opt][0]=='-')) {
		var sw = phantom.args[opt];
		switch(sw) {
			case '-a1':
				opt++;
				a1 = phantom.args[opt];
				break;
			case '-a2':
				opt++;
				a2 = phantom.args[opt];
				break;
			case '-a3':
				opt++;
				a3 = phantom.args[opt];
				break;
			case '-a4':
				opt++;
				a4 = phantom.args[opt];
				break;
			default:
				console.log('Unknown switch: ' + phantom.args[opt]);
				phantom.exit();
				break;
		}
		opt++;
	}
}

This can easily be modified to work with an array of argument results or you can simply read in each argument into its own variable. Also, you can read in integers and in your application logic, use isNaN() to check if the input is a valid integer.

Progress on my Xbox Live data scraper

11. October 2011 13:34 by Cameron in PhantomJS, Xbox Live  //  Tags: , , , , , ,   //   Comments

After many hours working with PhantomJS, I have finally managed to get all available data from xbox.com for my Xbox Live profile using jQuery selectors. The data is outputted as xml and will be imported into my database for quick retrieval and record of historical data. Here is what the xml for my profile looks like:

<xboxGamer>
  <profile gamertag='pcmantinker'>
    <gamerScore>545</gamerScore>
    <avatarSmall>http://avatar.xboxlive.com/avatar/pcmantinker/avatarpic-s.png</avatarSmall>
    <avatarLarge>http://avatar.xboxlive.com/avatar/pcmantinker/avatarpic-l.png</avatarLarge>
    <avatarBody>http://avatar.xboxlive.com/avatar/pcmantinker/avatar-body.png</avatarBody>
    <location>Louisiana</location>
    <bio>Hey guys,

I'm a senior in Computer Science at LSU. I love to play video games, compose piano music, and work on programming projects during my spare time.</bio>
    <motto>Geaux Tigers!</motto>
  </profile>
  <games>
    <game name='DiRT 2'>
      <link>http://marketplace.xbox.com/en-US/Title/1129121824</link>
      <smallImage>http://tiles.xbox.com/consoleAssets/434D0820/en-US/smallboxart.jpg</smallImage>
      <largeImage>http://tiles.xbox.com/consoleAssets/434D0820/en-US/largeboxart.jpg</largeImage>
      <lastPlayed>1/29/2011</lastPlayed>
      <earnedGamerScore>320</earnedGamerScore>
      <totalGamerScore>1000</totalGamerScore>
      <earnedAchievements>21</earnedAchievements>
      <totalAchievements>47</totalAchievements>
      <percentComplete>44</percentComplete>
      <achievementsLink>http://live.xbox.com/en-US/GameCenter/Achievements?titleId=1129121824</achievementsLink>
      <achievements>
        <achievement name='Jet Setter'>
          <description>You've raced in every location. You'll need a new passport soon!</description>
          <image>http://tiles.xbox.com/tiles/Yz/y8/1oCLiGJhbC8wCxsDGyxTWTIwL2FjaC8wLzFEAAAAAOfn5-mTPH8=.jpg</image>
          <gamerScore>15</gamerScore>
          <achievedOn>1/29/2011</achievedOn>
        </achievement>
        <achievement name='All-Star'>
          <description>You're an All-Star driver now!</description>
          <image>http://tiles.xbox.com/tiles/c9/9G/0ICLiGJhbC9CCxsDGyxTWTIwL2FjaC8wLzE2AAAAAOfn5-9p328=.jpg</image>
          <gamerScore>25</gamerScore>
          <achievedOn>1/1/2011</achievedOn>
        </achievement>
        <achievement name="Aren't You Popular!">
          <description>You're pretty popular! You're true friends with four superstars.</description>
          <image>http://tiles.xbox.com/tiles/Pz/3F/14CLiGJhbC9GCxsDGyxTWTIwL2FjaC8wLzEyAAAAAOfn5-jqPSM=.jpg</image>
          <gamerScore>15</gamerScore>
          <achievedOn>1/1/2011</achievedOn>
        </achievement>
        <achievement name='Just Drive!'>
          <description>You drove 100 miles (161 kilometres) without unlocking any other achievement.</description>
          <image>http://tiles.xbox.com/tiles/sP/wW/1oCLiGJhbC9ECBsDGyxTWTIwL2FjaC8wLzIwAAAAAOfn5-k5-Kw=.jpg</image>
          <gamerScore>15</gamerScore>
          <achievedOn>1/1/2011</achievedOn>
        </achievement>
        <achievement name='Terminated!'>
          <description>That's gotta hurt! You took terminal damage at over 140mph (225km/h).</description>
          <image>http://tiles.xbox.com/tiles/Fh/f9/1YCLiGJhbC82FQQcXCdRFzIwL2FjaC8wL0IAAAAA5+fn+tIXDQ==.jpg</image>
          <gamerScore>15</gamerScore>
          <achievedOn>8/14/2010</achievedOn>
        </achievement>
        <achievement name='Big Bucks!'>
          <description>Money! You've earned those Big Bucks!</description>
          <image>http://tiles.xbox.com/tiles/Sh/aE/1ICLiGJhbC9AFQQcXCdRFzIwL2FjaC8wLzQAAAAA5+fn+6sWUQ==.jpg</image>
          <gamerScore>25</gamerScore>
          <achievedOn>8/14/2010</achievedOn>
        </achievement>
        <achievement name='X Games Asia Champ'>
          <description>Banzai! You've won X Games Asia.</description>
          <image>http://tiles.xbox.com/tiles/7P/1v/14CLiGJhbC8yCBsDGyxTWTIwL2FjaC8wLzJGAAAAAOfn5-hA-fA=.jpg</image>
          <gamerScore>25</gamerScore>
          <achievedOn>8/14/2010</achievedOn>
        </achievement>
        <achievement name='Gate Crashed!'>
          <description>Every gate! That's sick! You hit every gate in a Gatecrasher event. </description>
          <image>http://tiles.xbox.com/tiles/Gc/wE/1ICLiGJhbC9ECxsDGyxTWTIwL2FjaC8wLzEwAAAAAOfn5-srzAU=.jpg</image>
          <gamerScore>10</gamerScore>
          <achievedOn>8/14/2010</achievedOn>
        </achievement>
        <achievement name='Friends For Life'>
          <description>You became true friends with a superstar!</description>
          <image>http://tiles.xbox.com/tiles/ij/Tk/1YCLiGJhbC9FCxsDGyxTWTIwL2FjaC8wLzExAAAAAOfn5-rLNJY=.jpg</image>
          <gamerScore>10</gamerScore>
          <achievedOn>8/11/2010</achievedOn>
        </achievement>
        <achievement name='Block Party'>
          <description>You're a Blockbuster. You've beaten Ken in a Throwdown.</description>
          <image>http://tiles.xbox.com/tiles/mt/ce/04CLiGJhbC83CxsDGyxTWTIwL2FjaC8wLzFDAAAAAOfn5-wx14Y=.jpg</image>
          <gamerScore>15</gamerScore>
          <achievedOn>8/10/2010</achievedOn>
        </achievement>
        <achievement name='Piece Of Cake'>
          <description>One challenge down! You've won a Throwdown.</description>
          <image>http://tiles.xbox.com/tiles/4C/em/0YCLiGJhbC9DCxsDGyxTWTIwL2FjaC8wLzE3AAAAAOfn5-6JJ-w=.jpg</image>
          <gamerScore>10</gamerScore>
          <achievedOn>8/10/2010</achievedOn>
        </achievement>
        <achievement name='That All You Got?'>
          <description>You beat a Superstar's time in a Timed Throwdown.</description>
          <image>http://tiles.xbox.com/tiles/1P/RY/04CLiGJhbC9GCRsDGyxTWTIwL2FjaC8wLzMyAAAAAOfn5-x39Mg=.jpg</image>
          <gamerScore>15</gamerScore>
          <achievedOn>8/10/2010</achievedOn>
        </achievement>
        <achievement name='Head Over Wheels'>
          <description>That's some hard charging! You rolled your vehicle but still won the race.</description>
          <image>http://tiles.xbox.com/tiles/8Q/uC/1YCLiGJhbC9MFQQcXCdRFzIwL2FjaC8wLzgAAAAA5+fn+q0L6g==.jpg</image>
          <gamerScore>10</gamerScore>
          <achievedOn>8/10/2010</achievedOn>
        </achievement>
        <achievement name='Getting A Reputation'>
          <description>You're getting a reputation out there. You've reached Level 15.</description>
          <image>http://tiles.xbox.com/tiles/fc/th/04CLiGJhbC9NCxsDGyxTWTIwL2FjaC8wLzE5AAAAAOfn5-xOy2E=.jpg</image>
          <gamerScore>15</gamerScore>
          <achievedOn>8/9/2010</achievedOn>
        </achievement>
        <achievement name='Gut It Out!'>
          <description>It’s the result that counts! You won an event without winning a single race.</description>
          <image>http://tiles.xbox.com/tiles/-x/+l/1oCLiGJhbC9DFQQcXCdRFzIwL2FjaC8wLzcAAAAA5+fn+Yof5A==.jpg</image>
          <gamerScore>10</gamerScore>
          <achievedOn>8/9/2010</achievedOn>
        </achievement>
        <achievement name='Pro'>
          <description>You've proved yourself in the Pro division. Now take it to the next level.</description>
          <image>http://tiles.xbox.com/tiles/xt/Zn/0oCLiGJhbC9BCxsDGyxTWTIwL2FjaC8wLzE1AAAAAOfn5-1I1to=.jpg</image>
          <gamerScore>15</gamerScore>
          <achievedOn>8/9/2010</achievedOn>
        </achievement>
        <achievement name='Keep It Clean!'>
          <description>This game has damage? You won a race without crashing!</description>
          <image>http://tiles.xbox.com/tiles/ox/7c/14CLiGJhbC81FQQcXCdRFzIwL2FjaC8wL0EAAAAA5+fn+PMeuA==.jpg</image>
          <gamerScore>20</gamerScore>
          <achievedOn>8/6/2010</achievedOn>
        </achievement>
        <achievement name='X Games Europe Champ'>
          <description>Blimey! You've won X Games Europe.</description>
          <image>http://tiles.xbox.com/tiles/Wf/RO/1YCLiGJhbC8xCBsDGyxTWTIwL2FjaC8wLzJFAAAAAOfn5-ph9EU=.jpg</image>
          <gamerScore>20</gamerScore>
          <achievedOn>8/5/2010</achievedOn>
        </achievement>
        <achievement name='Rookie Coming Through!'>
          <description>You gotta start somewhere. You've raced in your first DiRT Tour race.</description>
          <image>http://tiles.xbox.com/tiles/VS/6H/04CLiGJhbC9ACxsDGyxTWTIwL2FjaC8wLzE0AAAAAOfn5-yoLkk=.jpg</image>
          <gamerScore>10</gamerScore>
          <achievedOn>6/25/2010</achievedOn>
        </achievement>
        <achievement name='Nailed an Event'>
          <description>You've won an event. Now it's time to take on the world.</description>
          <image>http://tiles.xbox.com/tiles/he/8d/1ICLiGJhbC83FQQcXCdRFzIwL2FjaC8wL0MAAAAA5+fn+zLvng==.jpg</image>
          <gamerScore>10</gamerScore>
          <achievedOn></achievedOn>
        </achievement>
        <achievement name='Gnarly!'>
          <description>Flashy finish! You crossed the finish line with no more than two wheels touching the ground!</description>
          <image>http://tiles.xbox.com/tiles/Yv/Ni/1ICLiGJhbC9NFQQcXCdRFzIwL2FjaC8wLzkAAAAA5+fn+03zeQ==.jpg</image>
          <gamerScore>15</gamerScore>
          <achievedOn></achievedOn>
        </achievement>
        <achievement name='Dedication'>
          <description>Win every event in the DiRT Tour.</description>
          <image>http://tiles.xbox.com/tiles/2b/eV/0jc8P2NhbC9BFQQcXCdRFzIwL2FjaC8wLzUAAAABUFBQ-bq3wg==.jpg</image>
          <gamerScore>80</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='The Full Set'>
          <description>Drive every vehicle in the game.</description>
          <image>http://tiles.xbox.com/tiles/bL/60/0Dc8P2NhbC9CFQQcXCdRFzIwL2FjaC8wLzYAAAABUFBQ-5u+dw==.jpg</image>
          <gamerScore>30</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name="You're The Best, Around!">
          <description>Win a race at every track in the game (in any game mode).</description>
          <image>http://tiles.xbox.com/tiles/fF/1O/1jc8P2NhbC8wFQQcXCdRFzIwL2FjaC8wL0QAAAABUFBQ+WFdZw==.jpg</image>
          <gamerScore>20</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='Dominated!'>
          <description>Win a domination event, getting the maximum available points.</description>
          <image>http://tiles.xbox.com/tiles/76/Wu/1zc8P2NhbC8xFQQcXCdRFzIwL2FjaC8wL0UAAAABUFBQ+IGl9A==.jpg</image>
          <gamerScore>15</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='On The Bubble'>
          <description>Be second to last at each elimination point but go on to win a Last Man Standing race.</description>
          <image>http://tiles.xbox.com/tiles/Wq/yP/1Tc8P2NhbC8yFQQcXCdRFzIwL2FjaC8wL0YAAAABUFBQ+qCsQQ==.jpg</image>
          <gamerScore>15</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='Best Friends Forever!'>
          <description>Become true friends with every superstar.</description>
          <image>http://tiles.xbox.com/tiles/rJ/zU/0Tc8P2NhbC9HCxsDGyxTWTIwL2FjaC8wLzEzAAAAAVBQUP77nLA=.jpg</image>
          <gamerScore>30</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='I like a Challenge'>
          <description>Win 12 Throwdowns.</description>
          <image>http://tiles.xbox.com/tiles/7m/pw/1Tc8P2NhbC9MCxsDGyxTWTIwL2FjaC8wLzE4AAAAAVBQUPpfavI=.jpg</image>
          <gamerScore>20</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='As Good As It Gets'>
          <description>Reach Level 30 in the DiRT Tour.</description>
          <image>http://tiles.xbox.com/tiles/vH/8u/1zc8P2NhbC81CxsDGyxTWTIwL2FjaC8wLzFBAAAAAVBQUPgBf6A=.jpg</image>
          <gamerScore>30</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='Whose Backyard?'>
          <description>Beat Travis Pastrana in a Throwdown.</description>
          <image>http://tiles.xbox.com/tiles/CX/YP/1Tc8P2NhbC82CxsDGyxTWTIwL2FjaC8wLzFCAAAAAVBQUPogdhU=.jpg</image>
          <gamerScore>15</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='World Tour Winner'>
          <description>Win all the World Tour events.</description>
          <image>http://tiles.xbox.com/tiles/8J/2t/0Dc8P2NhbC8xCxsDGyxTWTIwL2FjaC8wLzFFAAAAAVBQUP+Cnew=.jpg</image>
          <gamerScore>50</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='X Games Champ!'>
          <description>Win all the X Games events.</description>
          <image>http://tiles.xbox.com/tiles/RZ/SM/0jc8P2NhbC8yCxsDGyxTWTIwL2FjaC8wLzFGAAAAAVBQUP2jlFk=.jpg</image>
          <gamerScore>50</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='Secret Achievement'>
          <description>Continue playing to unlock this secret achievement.</description>
          <image>/Content/Images/HiddenAchievement.png</image>
          <gamerScore>75</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='Five Of Your Best'>
          <description>Win 5 Pro Tour races over Games for Windows - LIVE.</description>
          <image>http://tiles.xbox.com/tiles/ll/Qm/0jc8P2NhbC9GCBsDGyxTWTIwL2FjaC8wLzIyAAAAAVBQUP0JVIo=.jpg</image>
          <gamerScore>15</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='Tourney'>
          <description>Take part in four tournaments.</description>
          <image>http://tiles.xbox.com/tiles/Ba/zG/0zc8P2NhbC9HCBsDGyxTWTIwL2FjaC8wLzIzAAAAAVBQUPzprBk=.jpg</image>
          <gamerScore>20</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='Famous'>
          <description>Reach Online Fame Level 15.</description>
          <image>http://tiles.xbox.com/tiles/-E/dk/1jc8P2NhbC9ACBsDGyxTWTIwL2FjaC8wLzI0AAAAAVBQUPlLR+A=.jpg</image>
          <gamerScore>20</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='Notorious'>
          <description>Reach Online Fame Level 30.</description>
          <image>http://tiles.xbox.com/tiles/b7/+E/1zc8P2NhbC9BCBsDGyxTWTIwL2FjaC8wLzI1AAAAAVBQUPirv3M=.jpg</image>
          <gamerScore>40</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='Rush Hour'>
          <description>Win a Pro Tour race against 7 competitors over Games for Windows - LIVE.</description>
          <image>http://tiles.xbox.com/tiles/2r/al/1Tc8P2NhbC9CCBsDGyxTWTIwL2FjaC8wLzI2AAAAAVBQUPqKtsY=.jpg</image>
          <gamerScore>10</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='Mix It Up!'>
          <description>Play in every discipline over Games for Windows - LIVE (Pro Tour or Jam Session).</description>
          <image>http://tiles.xbox.com/tiles/SU/5F/1Dc8P2NhbC9DCBsDGyxTWTIwL2FjaC8wLzI3AAAAAVBQUPtqTlU=.jpg</image>
          <gamerScore>15</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='In His Footsteps'>
          <description>Win 25 Rally races in the DiRT Tour.</description>
          <image>http://tiles.xbox.com/tiles/R1/pi/1zc8P2NhbC9MCBsDGyxTWTIwL2FjaC8wLzI4AAAAAVBQUPhNWls=.jpg</image>
          <gamerScore>25</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='With Great Honor'>
          <description>Achieve a 'Cautious' impact rating in Multiplayer races (Pro Tour or Jam Session).</description>
          <image>http://tiles.xbox.com/tiles/1K/KC/1jc8P2NhbC9NCBsDGyxTWTIwL2FjaC8wLzI5AAAAAVBQUPmtosg=.jpg</image>
          <gamerScore>15</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='Mirra Image'>
          <description>Beat Dave Mirra in a Throwdown.</description>
          <image>http://tiles.xbox.com/tiles/FU/88/1Tc8P2NhbC81CBsDGyxTWTIwL2FjaC8wLzJBAAAAAVBQUPoTTwk=.jpg</image>
          <gamerScore>15</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name="I Ain't Afraid Of No Ghost!">
          <description>Choose a ghost, download and beat it!</description>
          <image>http://tiles.xbox.com/tiles/oE/Yd/1zc8P2NhbC82CBsDGyxTWTIwL2FjaC8wLzJCAAAAAVBQUPgyRrw=.jpg</image>
          <gamerScore>10</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='Dialled In'>
          <description>Win a race using a custom vehicle setup.</description>
          <image>http://tiles.xbox.com/tiles/M7/79/1jc8P2NhbC83CBsDGyxTWTIwL2FjaC8wLzJDAAAAAVBQUPnSvi8=.jpg</image>
          <gamerScore>5</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='Two Cups One Girl'>
          <description>Win two team events with the same female team-mate.</description>
          <image>http://tiles.xbox.com/tiles/yl/Vf/0zc8P2NhbC8wCBsDGyxTWTIwL2FjaC8wLzJEAAAAAVBQUPxwVdY=.jpg</image>
          <gamerScore>10</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='X Games America Champ'>
          <description>Win X Games America.</description>
          <image>http://tiles.xbox.com/tiles/8l/xo/1zc8P2NhbC9ECRsDGyxTWTIwL2FjaC8wLzMwAAAAAVBQUPhHXO4=.jpg</image>
          <gamerScore>30</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='Showboating'>
          <description>Bask in your success at X Games America.</description>
          <image>http://tiles.xbox.com/tiles/Ya/SI/1jc8P2NhbC9FCRsDGyxTWTIwL2FjaC8wLzMxAAAAAVBQUPmnpH0=.jpg</image>
          <gamerScore>20</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
      </achievements>
    </game>
    <game name='Halo 2 (PC)'>
      <link>http://marketplace.xbox.com/en-US/Title/1297287183</link>
      <smallImage>http://tiles.xbox.com/consoleAssets/4D53080F/en-US/smallboxart.jpg</smallImage>
      <largeImage>http://tiles.xbox.com/consoleAssets/4D53080F/en-US/largeboxart.jpg</largeImage>
      <lastPlayed>7/3/2010</lastPlayed>
      <earnedGamerScore>225</earnedGamerScore>
      <totalGamerScore>1000</totalGamerScore>
      <earnedAchievements>12</earnedAchievements>
      <totalAchievements>41</totalAchievements>
      <percentComplete>29</percentComplete>
      <achievementsLink>http://live.xbox.com/en-US/GameCenter/Achievements?titleId=1297287183</achievementsLink>
      <achievements>
        <achievement name='Cairo Station'>
          <description>Complete Cairo Station at Normal, Heroic, or Legendary</description>
          <image>http://tiles.xbox.com/tiles/4D/OV/14CLiGJhbC9BFQRrXVBRFzBGL2FjaC8wLzUAAAAA5+fn+Loz+w==.jpg</image>
          <gamerScore>30</gamerScore>
          <achievedOn>7/3/2010</achievedOn>
        </achievement>
        <achievement name='Assassin'>
          <description>Melee an opponent from behind</description>
          <image>http://tiles.xbox.com/tiles/ii/DX/04CLiGJhbC9HFQRrXVBRFzBGL2FjaC8wLzMAAAAA5+fn-PggkQ==.jpg</image>
          <gamerScore>10</gamerScore>
          <achievedOn>9/10/2008</achievedOn>
        </achievement>
        <achievement name='Sniper Kill'>
          <description>Kill an opponent with the sniper rifle or the beam rifle</description>
          <image>http://tiles.xbox.com/tiles/3k/E6/14CLiGJhbC9FCBt0GltTWTBGL2FjaC8wLzIxAAAAAOfn5-gVQcI=.jpg</image>
          <gamerScore>10</gamerScore>
          <achievedOn>9/10/2008</achievedOn>
        </achievement>
        <achievement name='Bonecracker'>
          <description>Hit and kill an opponent with a melee attack</description>
          <image>http://tiles.xbox.com/tiles/c8/t1/1oCLiGJhbC9AFQRrXVBRFzBGL2FjaC8wLzQAAAAA5+fn+VrLaA==.jpg</image>
          <gamerScore>10</gamerScore>
          <achievedOn>9/10/2008</achievedOn>
        </achievement>
        <achievement name='Running Riot'>
          <description>Kill 10 opponents in a row in the same game, without dying</description>
          <image>http://tiles.xbox.com/tiles/nn/lw/1oCLiGJhbC8wCxt0GltTWTBGL2FjaC8wLzFEAAAAAOfn5-lfeYI=.jpg</image>
          <gamerScore>20</gamerScore>
          <achievedOn>9/10/2008</achievedOn>
        </achievement>
        <achievement name='Killing Spree'>
          <description>Kill 5 opponents in a row in the same game, without dying</description>
          <image>http://tiles.xbox.com/tiles/d3/Eo/1YCLiGJhbC9FCxt0GltTWTBGL2FjaC8wLzExAAAAAOfn5-oHcWs=.jpg</image>
          <gamerScore>20</gamerScore>
          <achievedOn>9/10/2008</achievedOn>
        </achievement>
        <achievement name='The Arbiter'>
          <description>Complete The Arbiter at Normal, Heroic, or Legendary</description>
          <image>http://tiles.xbox.com/tiles/Gd/g3/0oCLiGJhbC9GFQRrXVBRFzBGL2FjaC8wLzIAAAAA5+fn-RjYAg==.jpg</image>
          <gamerScore>30</gamerScore>
          <achievedOn>9/5/2008</achievedOn>
        </achievement>
        <achievement name='Stick It'>
          <description>Stick an opponent with a plasma grenade and blow them up</description>
          <image>http://tiles.xbox.com/tiles/a0/gb/1YCLiGJhbC9GCBt0GltTWTBGL2FjaC8wLzIyAAAAAOfn5-o0SHc=.jpg</image>
          <gamerScore>5</gamerScore>
          <achievedOn>9/4/2008</achievedOn>
        </achievement>
        <achievement name='Roadkill'>
          <description>Run over, hit and kill an opponent with a vehicle</description>
          <image>http://tiles.xbox.com/tiles/Z5/LS/04CLiGJhbC83Cxt0GltTWTBGL2FjaC8wLzFDAAAAAOfn5-z9kns=.jpg</image>
          <gamerScore>10</gamerScore>
          <achievedOn>9/4/2008</achievedOn>
        </achievement>
        <achievement name='Flaming Ninja'>
          <description>Kill an opponent that has the Ninja achievement</description>
          <image>http://tiles.xbox.com/tiles/vD/Ls/1oCLiGJhbC83FQRrXVBRFzBGL2FjaC8wL0MAAAAA5+fn+cMypw==.jpg</image>
          <gamerScore>20</gamerScore>
          <achievedOn>9/3/2008</achievedOn>
        </achievement>
        <achievement name='Metropolis'>
          <description>Complete Metropolis at Normal, Heroic, or Legendary</description>
          <image>http://tiles.xbox.com/tiles/O5/Or/0oCLiGJhbC9BCxt0GltTWTBGL2FjaC8wLzE1AAAAAOfn5-2Ekyc=.jpg</image>
          <gamerScore>30</gamerScore>
          <achievedOn>9/3/2008</achievedOn>
        </achievement>
        <achievement name='Outskirts'>
          <description>Complete Outskirts at Normal, Heroic, or Legendary</description>
          <image>http://tiles.xbox.com/tiles/E3/ZN/0oCLiGJhbC9MCxt0GltTWTBGL2FjaC8wLzE4AAAAAOfn5-1idg8=.jpg</image>
          <gamerScore>30</gamerScore>
          <achievedOn>9/3/2008</achievedOn>
        </achievement>
        <achievement name='Oracle'>
          <description>Complete The Oracle at Normal, Heroic, or Legendary</description>
          <image>http://tiles.xbox.com/tiles/HT/ub/1jc8P2NhbC9DCxt0GltTWTBGL2FjaC8wLzE3AAAAAVBQUPm0OwE=.jpg</image>
          <gamerScore>30</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='Delta Halo'>
          <description>Complete Delta Halo at Normal, Heroic, or Legendary</description>
          <image>http://tiles.xbox.com/tiles/W3/di/0Tc8P2NhbC9NFQRrXVBRFzBGL2FjaC8wLzkAAAABUFBQ-k13QA==.jpg</image>
          <gamerScore>30</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='Regret'>
          <description>Complete Regret at Normal, Heroic, or Legendary</description>
          <image>http://tiles.xbox.com/tiles/9D/PD/1Tc8P2NhbC82Cxt0GltTWTBGL2FjaC8wLzFCAAAAAVBQUPrsM+g=.jpg</image>
          <gamerScore>30</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='Sacred Icon'>
          <description>Complete Sacred Icon at Normal, Heroic, or Legendary</description>
          <image>http://tiles.xbox.com/tiles/Dd/hh/0Dc8P2NhbC8xCxt0GltTWTBGL2FjaC8wLzFFAAAAAVBQUP9O2BE=.jpg</image>
          <gamerScore>30</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='Quarantine Zone'>
          <description>Complete Quarantine Zone at Normal, Heroic, or Legendary</description>
          <image>http://tiles.xbox.com/tiles/gN/dc/1Dc8P2NhbC9NCxt0GltTWTBGL2FjaC8wLzE5AAAAAVBQUPtz15w=.jpg</image>
          <gamerScore>30</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='Gravemind'>
          <description>Complete Gravemind at Normal, Heroic, or Legendary</description>
          <image>http://tiles.xbox.com/tiles/1n/hf/1Tc8P2NhbC8xFQRrXVBRFzBGL2FjaC8wL0UAAAABUFBQ+nB4zQ==.jpg</image>
          <gamerScore>30</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='Uprising'>
          <description>Complete Uprising at Normal, Heroic, or Legendary</description>
          <image>http://tiles.xbox.com/tiles/kv/pI/1zc8P2NhbC9BCBt0GltTWTBGL2FjaC8wLzI1AAAAAVBQUPhn+o4=.jpg</image>
          <gamerScore>30</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='High Charity'>
          <description>Complete High Charity at Normal, Heroic, or Legendary</description>
          <image>http://tiles.xbox.com/tiles/Ke/dO/1jc8P2NhbC9NCBt0GltTWTBGL2FjaC8wLzI5AAAAAVBQUPlh5zU=.jpg</image>
          <gamerScore>30</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='The Great Journey'>
          <description>Complete The Great Journey at Normal, Heroic, or Legendary</description>
          <image>http://tiles.xbox.com/tiles/+O/kK/0zc8P2NhbC9HCBt0GltTWTBGL2FjaC8wLzIzAAAAAVBQUPwl6eQ=.jpg</image>
          <gamerScore>30</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='Warrior'>
          <description>Complete the game on Normal difficulty</description>
          <image>http://tiles.xbox.com/tiles/uh/+u/1zc8P2NhbC9MCBt0GltTWTBGL2FjaC8wLzI4AAAAAVBQUPiBH6Y=.jpg</image>
          <gamerScore>100</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='Hero'>
          <description>Complete the game on Heroic difficulty</description>
          <image>http://tiles.xbox.com/tiles/Y3/F+/1zc8P2NhbC8yFQRrXVBRFzBGL2FjaC8wL0YAAAABUFBQ+FFxeA==.jpg</image>
          <gamerScore>50</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='Legend'>
          <description>Complete the game on Legendary difficulty</description>
          <image>http://tiles.xbox.com/tiles/qD/K6/1Dc8P2NhbC9ACxt0GltTWTBGL2FjaC8wLzE0AAAAAVBQUPuVMrQ=.jpg</image>
          <gamerScore>50</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='King of the Scarab'>
          <description>Acquire the Scarab gun on the Metropolis level, any difficulty</description>
          <image>http://tiles.xbox.com/tiles/Ud/kY/0Tc8P2NhbC9HCxt0GltTWTBGL2FjaC8wLzEzAAAAAVBQUP432U0=.jpg</image>
          <gamerScore>20</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='Silent but Deadly'>
          <description>Kill 7 opponents from behind in a row without being spotted, any level, any difficulty</description>
          <image>http://tiles.xbox.com/tiles/uN/FA/0jc8P2NhbC8yCxt0GltTWTBGL2FjaC8wLzFGAAAAAVBQUP1v0aQ=.jpg</image>
          <gamerScore>20</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='Demon'>
          <description>Complete any campaign level without dying once on Normal, Heroic, or Legendary</description>
          <image>http://tiles.xbox.com/tiles/mp/rc/0jc8P2NhbC81FQRrXVBRFzBGL2FjaC8wL0EAAAABUFBQ-fOagQ==.jpg</image>
          <gamerScore>25</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='Go Ape Shiv'>
          <description>Kill an enraged, berserk Brute by melee, any level, any difficulty</description>
          <image>http://tiles.xbox.com/tiles/RY/C-/1Dc8P2NhbC8wFQRrXVBRFzBGL2FjaC8wL0QAAAABUFBQ+5CAXg==.jpg</image>
          <gamerScore>20</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='Counterpoint'>
          <description>Kill the sword carrier</description>
          <image>http://tiles.xbox.com/tiles/xp/ul/0zc8P2NhbC9DFQRrXVBRFzBGL2FjaC8wLzcAAAABUFBQ-Iqb3Q==.jpg</image>
          <gamerScore>5</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='Carjacking'>
          <description>Steal an occupied vehicle from an opponent and take it over</description>
          <image>http://tiles.xbox.com/tiles/VW/NF/0jc8P2NhbC9CFQRrXVBRFzBGL2FjaC8wLzYAAAABUFBQ-WpjTg==.jpg</image>
          <gamerScore>5</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='Violent Cartographer'>
          <description>Play every available default map and stay in each game until it’s over</description>
          <image>http://tiles.xbox.com/tiles/tA/uJ/1Dc8P2NhbC9DCBt0GltTWTBGL2FjaC8wLzI3AAAAAVBQUPumC6g=.jpg</image>
          <gamerScore>25</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='Rainman'>
          <description>Play every available game variant, with at least 3 other players, to the end of each game</description>
          <image>http://tiles.xbox.com/tiles/QT/ri/1zc8P2NhbC81Cxt0GltTWTBGL2FjaC8wLzFBAAAAAVBQUPjNOl0=.jpg</image>
          <gamerScore>25</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='Double Kill'>
          <description>Kill 2 opponents within 4 seconds</description>
          <image>http://tiles.xbox.com/tiles/L5/P9/0Dc8P2NhbC82FQRrXVBRFzBGL2FjaC8wL0IAAAABUFBQ-9KTNA==.jpg</image>
          <gamerScore>20</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='Triple Kill'>
          <description>Kill 3 opponents within 4 seconds</description>
          <image>http://tiles.xbox.com/tiles/AQ/Ko/1jc8P2NhbC9ACBt0GltTWTBGL2FjaC8wLzI0AAAAAVBQUPmHAh0=.jpg</image>
          <gamerScore>20</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='Killtacular'>
          <description>Kill 4 opponents within 4 seconds</description>
          <image>http://tiles.xbox.com/tiles/wi/H4/0Dc8P2NhbC9GCxt0GltTWTBGL2FjaC8wLzEyAAAAAVBQUP-XId4=.jpg</image>
          <gamerScore>20</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='Skewer Stopper'>
          <description>Kill the sword carrier after they kill 5 or more times in a row with the sword</description>
          <image>http://tiles.xbox.com/tiles/Te/Ar/0Tc8P2NhbC9ECBt0GltTWTBGL2FjaC8wLzIwAAAAAVBQUP4E4FE=.jpg</image>
          <gamerScore>20</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='Vigilante'>
          <description>Stop a Killing Spree of another player</description>
          <image>http://tiles.xbox.com/tiles/J-/Np/1Tc8P2NhbC9CCBt0GltTWTBGL2FjaC8wLzI2AAAAAVBQUPpG8zs=.jpg</image>
          <gamerScore>10</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='Air Traffic Control'>
          <description>Blow up the Banshee in flight while manned, with grenades or the rocket launcher</description>
          <image>http://tiles.xbox.com/tiles/rI/jn/1zc8P2NhbC9FFQRrXVBRFzBGL2FjaC8wLzEAAAABUFBQ+MiItw==.jpg</image>
          <gamerScore>20</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='Decorated Soldier'>
          <description>Get awarded at least 8 different medals in one non-team game</description>
          <image>http://tiles.xbox.com/tiles/yI/+C/0Dc8P2NhbC9MFQRrXVBRFzBGL2FjaC8wLzgAAAABUFBQ-62P0w==.jpg</image>
          <gamerScore>10</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='Ninja'>
          <description>In a non-team game, kill 5 people by melee, from behind</description>
          <image>http://tiles.xbox.com/tiles/js/N7/1zc8P2NhbC9CCxt0GltTWTBGL2FjaC8wLzE2AAAAAVBQUPhUw5I=.jpg</image>
          <gamerScore>20</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
        <achievement name='Hired Gun'>
          <description>Kill an opponent who has the Legend achievement</description>
          <image>http://tiles.xbox.com/tiles/5N/A5/0zc8P2NhbC9ECxt0GltTWTBGL2FjaC8wLzEwAAAAAVBQUPwW0Pg=.jpg</image>
          <gamerScore>20</gamerScore>
          <achievedOn>Not Achieved</achievedOn>
        </achievement>
      </achievements>
    </game>
  </games>
</xboxGamer>

As you can tell, the xml file for a user can be quite large. I've only got two games on Xbox Live currently, but some people may have upwards of 30+ games and each game can have 40+ achievements. In the xml data, there will be some redundancies, but when I import the data into my database, I will make sure to check for uniqueness before adding pre-existing data. The scraper has many phases. First, the raw data import phase where I select the data using jQuery selectors and put the data into javascript objects. The next phase is generating xml from the objects. The last phase is importing the data into my SQL database so that data may be properly related to each user. For example, if two people have the same game, their game lists will simply point to the same game and not make another entry in the database. The same goes for achievements. All unique achievements that exist for each game will be added to the database. For each achievement that a user completes for a game, the system will add an entry that relates the user to the game and achievement.

Passing information to and from webpages in PhantomJS

27. September 2011 11:01 by Cameron in javascript, PhantomJS  //  Tags: , , , , ,   //   Comments

Recently, I needed a way to pass dynamic content to and from webpages using PhantomJS as part of writing my screen scraper. I need the scraper to follow dynamic sets of links and scrape the data from each page. Since a webpage's scope is currently sand boxed, I had to find a way to pass data to and from webpages. With the addition of the new filesystem module in PhantomJS 1.3, it is now possible to pass data from the main scope to an individual page's scope. Any data that you want passed to a particular page should be saved as a javascript string to a javascript file. Then you can inject the javascript into the page on page.onLoadFinished so that the data is then accessible within the page's scope. For example:

var page = require('webpage').create(), 
     fs = require('fs'), 
     data = "var dataObject = { item: 'value' };", 
     fullpath;

fullpath = fs.workingDirectory + fs.separator + 'data.js';
// open file for writing
var dataFile = fs.open(fullpath, 'w');
dataFile.write(data);
dataFile.close();

// check that the file was successfully written
if(fs.size(fullpath) > 0) {
	console.log('File wrote successfully!');
	page.open('http://somesite.org/page.html');
	// put page data in a local variable
	var output = page.evaluate(function () {
		// print the output of the data object
		console.log(dataObject.item);
		return dataObject.item;
	});
	// output should be the same value as the page's dataObject.item
	console.log(output);
}
else {
	console.log('Error in writing the file!');
	phantom.exit();
}

page.onLoadFinished = function() {
	// inject the javascript data that we created earlier
	page.injectJS(fullpath);
}

For more information about PhantomJS' File System module, please visit: http://code.google.com/p/phantomjs/wiki/Interface#Filesystem_Module

While this solution may not be the best long term solution, it does provide a way to get data to and from your pages until official support for passing data to a webpage object becomes available in PhantomJS.

Take Screenshot of all HTML documents in a folder using PhantomJS

26. September 2011 01:14 by Cameron in javascript, PhantomJS, Programming  //  Tags: , , , ,   //   Comments

Recently I came across a question on stackoverflow that asked about how to take screenshots of all HTML files in a local folder. I have been playing with PhantomJS quite a bit lately and felt comfortable answering the question. Here is the code for those interested:

var page = require('webpage').create(), loadInProgress = false, fs = require('fs');
var htmlFiles = new Array();
console.log('working directory: ' + fs.workingDirectory);
var curdir = fs.list(fs.workingDirectory);

// loop through files and folders
for(var i = 0; i< curdir.length; i++)
{
	var fullpath = fs.workingDirectory + fs.separator + curdir[i];
	// check if item is a file
	if(fs.isFile(fullpath))
	{
		if(fullpath.indexOf('.html') != -1)
		{
			// show full path of file
			console.log('File path: ' + fullpath);
			htmlFiles.push(fullpath);
		}
	}
}

console.log('Number of Html Files: ' + htmlFiles.length);

// output pages as PNG
var pageindex = 0;

var interval = setInterval(function() {
	if (!loadInProgress && pageindex < htmlFiles.length) {
		console.log("image " + (pageindex + 1));
		page.open(htmlFiles[pageindex]);
	}
	if (pageindex == htmlFiles.length) {
		console.log("image render complete!");
		phantom.exit();
	}
}, 250);

page.onLoadStarted = function() {
	loadInProgress = true;
	console.log('page ' + (pageindex + 1) + ' load started');
};

page.onLoadFinished = function() {
	loadInProgress = false;
	page.render("images/output" + (pageindex + 1) + ".png");
	console.log('page ' + (pageindex + 1) + ' load finished');
	pageindex++;
}

The process is quite simple. First, I loop through all objects in the current working directory and check to see if each item is a file and whether it has the .html extension. Then I add each html file's filepath to an array that I later loop through to take the screenshots. A screenshot must be taken after the page is fully loaded so that the screenshot will contain actual content and not a blank image. This is done by saving the image on the page.onLoadFinished callback. The application loop for taking the screenshots inserts small 250ms delays between each request so that pages may fully load into the headless browser before advancing to the next page.

Month List

Tag cloud