XBox Live Data

20. September 2011 14:32 by Cameron in Programming, Xbox Live  //  Tags: , , , , , , , , , , ,   //   Comments

While my gaming social networking site, IGA: International Gamers' Alliance, is still under beta, I have been looking at ways to provide a more rich experience for my users. Lately I've been working on a way to gather data from XBox Live so that I can provide content to my users on IGA. I used to have a way to gather data from a RESTful API, using the official XBox Live API, that Microsoft employee, Duncan Mckenzie, used to host on his website. However, his service is no longer available. While there is an official XBox Live API, access to this API is restricted to those who are in the XBox Community Developer Program. Acceptance into this the XBCDP is very limited at the moment and it seems that only well known companies with sponsors receive membership into the program. 

While it would be very nice to get official access to the XBox Live API, it may be a while until I can get into the program. My social networking site, IGA, is still in beta and has much to be done on the roadmap to completion. Currently I am the only developer for the project and I am also in school so development is slow. Maybe once IGA is closer to completion, Microsoft will be more eager to accept me into the program. In the meantime, I have a solution for gathering data from XBox Live.

There are a couple of places to get data from XBox Live. There is the publicly available user's gamercard and the user's protected XBox.com profile. Getting data from the public gamercard is very easy. One could write a parser in PHP, C#, or even jQuery to get the different values from the HTML elements on the page. Retrieving data from a user's XBox.com profile requires a little more skill and resources. You cannot simply use cURL to remotely login to XBox.com since it has anti-bot mechanisms in place to check against the browser agent, browser cookies, and many other aspects that can't easily be manipulated with cURL. There is a remedy to this problem however.

This past summer, I learned about a headless webkit browser called PhantomJS from some co-workers while working on a project at work. We needed something that could run without a GUI on a server that could manipulate the DOM of a webpage. PhantomJS gave us exactly what we needed. After working on the project at work, it occurred to me that I could use PhantomJS in addition to jQuery to manipulate the DOM and screen scrape data from XBox.com.

I'm currently working on scripts to pull data from users' profiles including the users' games, the achievements earned in each game, and more information not publicly available on users' gamercards. Please understand though that screen scraping should only be done on a last resort and it is taxing on both ends to make numerous requests per day. I will implement some sort of data caching that will pull new data on a schedule to limit bandwidth usage. I plan to release this code to my Git hosting when it is finished. 

Gitting started with Git

15. August 2011 00:25 by Cameron in Git  //  Tags: , , , , , , , , ,   //   Comments

Git, created by Linus Torvalds, is a very high quality version control system. It was created with the task to manage the source tree of the Linux kernel. Torvalds didn't believe that pre-existing version control systems could give justice to the Linux kernel's source code given its massive size and collaborators so Torvalds created Git. If you are using other version control systems for your projects, consider reading this: http://whygitisbetterthanx.com/

This website explains the advantages in full of why Git is better than other version control systems available. 

Git is free and open source and is available for all platforms: Linux, Mac, Windows, Solaris, you name it

First, be sure to install git for your platform and then you can start playing around with different commands. Once you've installed git, here are a few references to get you started:  

http://book.git-scm.com/  

http://www.kernel.org/pub/software/scm/git/docs/  

Setting Up Git

In order to setup your environment for using a remote git repository, be sure to run these commands:

$ ssh-keygen -t rsa -C "youremail@site.com"

This command creates a public/private key pair for SSH. SSH is used by git to encrypt the connection to remote servers. When asked to where to save your public key, press enter. Then, when asked for a passphrase, leave it empty. Your screen should look like this:

Generating public/private rsa key pair.

Enter file in which to save the key (/home/cameron/.ssh/id_rsa): 

Created directory '/home/cameron/.ssh'.

Enter passphrase (empty for no passphrase): 

Enter same passphrase again: 

Your identification has been saved in /home/cameron/.ssh/id_rsa.

Your public key has been saved in /home/cameron/.ssh/id_rsa.pub.

After your public/private key have been setup, add your global user information:

$ git config --global user.name "Firstname Lastname"

$ git config --global user.email "your_email@youremail.com"

Now you are ready to clone a repository. If you run:

$ git clone git@git.tinksoft.net:test.git

A new directory will be created for the git repository, test, and all of the remote files in the repository will be downloaded into that directory.

Git Command Basics

A few common commands to git are cloning repositories, committing to repositories, pushing to repositories, and pulling from repositories. If you've worked with subversion before, "git clone" is like subversion checkout. It literally clones the remote repository in its current state to your local repository. However, "git commit" is not like subversion commit. When you commit to a git repository, you are only committing to your local repository until you push to the remote repository. Using "git push" is like subversion commit and will push your changes to the remote repository. On the first push, you need to run the command "git push origin <branch name>". This tells git to push the origin to the branch that you specify. After that first push, you can run "git push" thereafter. If you choose to switch branches later on, you simply need to run the original command and specify your origin branch. Similarly, "git pull" behaves like subversion update and pulls down changes from your remote repository into your locally cloned repository. The same  applies to the first "git pull" as does the first "git push". Git needs to know which branch to pull from.  

One thing about pushing and pulling is that if you are working in a team and multiple people are pushing and pulling to the remote repository, you may be required to pull before you push out your changes. Don't worry though. If you have a conflict with your changes, your code will not be overwritten. Git has a conflict resolution tool where you can choose which changes to accept. Another thing that is good practice is to always run "git status" before committing and pushing to your repository. This will allow you to confirm that you are indeed committing files that should be committed to your repository. Also, whatever shows up in a commit log will be pushed to your remote repository when you push our your changes. Be sure to only push out working code and not break the build for your team.

A few advanced commands include "git branch <branch name>" (branches the repository at its current state), "git merge -s ours <branch>" (merges a branch with current branch), and "git checkout <branch name>" (changes current working branch). Please be sure to read up on these commands so that you know how to use them correctly. In a project repository, you don't want to create unnecessary branches, merge branches incorrectly, or lose changes when switching branches. Another advanced topic is to create a .gitignore file for your repository and put all files that git should ignore into this file. Each file should be on a separate line. This can be helpful if you don't want files such as database configurations to be pushed to your remote repository. 

For more information about git, be sure to read the references I listed above and also check out some books on git for a more in depth discussion.

Month List

Tag cloud