Welcome to bpurcell.org, the personal homepage of Brandon Purcell. I started building bpurcell.org many, many years ago to share information I ran across everyday in my role as a Support Engineer and Consultant for Macromedia/Adobe. As a support engineer and consultant I always had a lot to blog about but as I moved into managment roles at Adobe my technical content dwindled.
I currently work as the Director of Technology for Universal Mind. My primary responsiblity is managing the SpatialKey project and it has been an amazing experience. I have been spending a lot of time working with Amazon Ec2 and will share my experiences through the blog in the future. I truly believe that Cloud Computing is the IT platform of the future and we have built the SpatialKey architecture on top of Ec2.
I was running out of space on a server the other day and was wondering what directory was full and what files I could clean up. There is a simple command that is helpful to identify usage by directory. This works from within any directory so you can walk the path from / down through your largest directories and look for files to clean up.
We use PostgreSQL for SpatialKey and because of that I have learned a lot about system administration with PostgreSQL. While debugging an issue with tables that had ballooned up I ran into this article which has some great tips for isolating db and table size with a few quick commands. http://www.thegeekstuff.com/2009/05/15-advanced-postgresql-commands-with-examples/
There is nothing more frustrating than taking 30 minutes filling out some text in a long form only to submit and have your session reset and lose all of your data. Here is a simple solution to keep a session open on forms where users may need an extended period of time for data entry. I included this code on any page where I need the session extended. You can remove the javascript include if JQuery is included on everyone of your pages already.
<!--- Include this file on any page where users may take a
long time filling in text and you want to keep the session open --->
<cfparam name="variables.refreshrate" default="60000">
<cfoutput>
<script type="text/javascript" src="/js/jquery-1.3.2.min.js"></script>
<script language="JavaScript" type="text/javascript">
$(document).ready(function(){
setTimeout("callserver()",#variables.refreshrate#);
});
function callserver()
{
var remoteURL = '/emptypage.cfm ';
$.get(remoteURL, function(data){
//alert(data);
setTimeout("callserver()",#variables.refreshrate#);
});
}
</script>
</cfoutput>
In the video we talk about Universal Mind, how SpatialKey came to be, a lot about what SpatialKey is and how it works. I dive into the architecture a bit and how we leveraged the cloud to build out our infrastructure.
Thanks to Jon and James hosting us, we crashed Jon's house for the video. I think Doug tossed back the most beers I can't keep up with him and still talk.
This issue drove me nuts today for several hours today, to be honest I was pretty close to breaking something! Earlier in the day I had the NFS mount working fine then I created an AMI and booted up another instance from the newly created AMI but in the new instance the mount kept failing. The error looks something like this:
[root@server]# mount -t nfs 192.168.2.1:/dbshare /mnt/dbshare
mount: 192.168.2.1:/dbshare failed, reason given by server: Permission denied
According to the error you would think that I have a configuration issue so I changed everything that I could think of within /etc/exports
My /etc/exports originally looked like this:
/dbshare 192.168.2.2(rw,sync) (where 192.168.2.2 is the client where I am performing the mount)
I changed it to something more open like this with no luck:
/dbshare 192.168.0.0/255.255.0.0(rw,sync)
I started looking around the logs on the server in /var/log/messages and found that it was authenticating fine Jun 11 19:04:00 servername mountd[5222]: authenticated mount request from 192.168.2.2:736 for /dbshare (/dbshare)
I was really frustrated at this point and I had already spent an hour on Google looking for the answer. I found another answer but the website was down, luckily the cached version on Google came to the rescue.
The ANSWER:
The problem was that the special nfsd file system that mounts to /proc/fs/nfsd wasn't mounted. I'm not sure how it gets mounted (maybe rc.sysinit does it?), but I tool the advice from the forum entry and added an entry to /etc/fstab
none /proc/fs/nfsd nfsd auto,defaults 0 0
then ran mount -a
After this the mount worked fine. I hope that someone finds this helpful.
I often need to copy code from a SVN repository to a location where I have files deployed or I may pull code from an SVN repository on the web and want to check it in locally. Both of these actions require the stripping out of all .svn directories spread throughout the directory structure. On Linux or Mac this is pretty simple with the following command
In recent months I have been using PHP to extend functionality inside of WordPress for the SpatialKey site and working with JQuery as well. After working with ColdFusion for so long I have gotten spoiled by the CFDUMP tag . Here are a few options similar to CFDUMP for both PHP and Javascript.
One challenge I have encountered with Amazon Ec2 is the sending of email from our web applications. If you try to send directly from sendmail or postfix then you might as well forget about guranteed delivery. A large amount of your email will end up in spam folders if it is even delivered at all.
There are a few problems with delivering email from Ec2
MX record will not map to your IP and you are using dynamic IPs (you can address this with elastic IP's) adding a SPF DNS record can help as well
Reverse DNS will map back to Amazon and not back to your Hostname
Many of the Ec2 IP's have been blacklisted due to abusers of the service sending spam.
There are a few solutions to this problem and I will propose two of them.
Using a google apps account:
If you are using a solution like Google Apps and have them host your email accounts then you can use gmail as your SMTP server. You will need to create an account donotreply@mydomain.com and use authentication in your applications to send the email. With Google apps you cannot override the "from" address when you send email it wil always become whatever you account you are sending from. For example if you create the account donotreply@mydomain.com and attempt to set the "from" in your code to send from brandon@mydomain.com google will override it and send from donotreply. The only option is to set "replyto" in your code and when a user replys they will send to your replyto account. With google apps you are limited to a maximum amount of 500 emails a day per account as well and if you are sending a lot of emails this can quickly become a problem. This is a great solution for small volumes of email and you delivery rates are very good.
Relay from localhost through a third party:
This blog post outlines a set of steps to relay through a local Postfix instance to a third party SMTP service. The great thing about this solution is that you can send email from your application to localhost without storing the authentication parameters in your applications code and have Postfix handle it all. If you have many applications sending email this can greatly simplify things. Also it allows your application to hand off the emails quickly to another service that can handle queuing in case the third party email service is down at any time. You could combine the approach above with this one but you would still have the 500 email limitation. I am searching for a good third party SMTP service that is reliable, the author of the blog post recommends AuthSMP. I have not tried them and their prices are not too high but not cheap either. I am going to do a little more digging and test some of the options and will report back to this posting.
This is common knowledge if you have been using Linux for a while but I still find it a helpful resource to understand how you set what programs are running when Linux starts. This is mainly specific to Red Hat or CentOS which I use on a regular basis.
Running level
Running level is the current running functional level of the operation system, from level 1 to 6, possessing different functions.
Here are the different running levels:
These levels are specified in the file /etc/inittab., which is the main file that the program init looks for, and the first running service is placed under the directory /etc/rc.d. For most Linux releases, startup scripts are all located in /etc/rc.d/init.d, which are all linked to the directory /etc/rc.d/rcn.d by ln command (here the n is the running level 0-6).
Setting services/applications to run at startup using chkconfig
chkconfig command (under redhat and centos)
Linux provides the command chkconfig to update and query system services of different running levels allowing you to set when certain process are started.
Syntax:
chkconfig --list [name]
chkconfig --add name
chkconfig --del name
chkconfig [--level levels] name
chkconfig [--level levels] name
chkconfig has five functions: add service, delete service, list service, change startup info and check the start state of specified service.
Option overview:
--level levels
specify running level, which is a character string composed of the number 0 to 7. For example:
--level 35 means to specify running level 3 and 5.
To stop the service nfs during running level 3,4,5, use the command next: chkconfig --level 345 nfs off
--add name
This option adds a new service, chkconfig ensures every running level an entrance to start (S) or to kill (K). if it is absent, then it would auto establish from default init script.
--del name
To delete service and delete related sign connections from /etc/rc[0-6].d.
--list name
List, if name is specified, then it only displays specified service name, otherwise, to list the state of all service at different running levels.
Usage examples:
As an example if you wanted mysql to run when the os starts you just need to do the following
/etc/init.d/mysqld must exist and needs to be an executable (chmod +x)
Add mysql - chkconfig --add mysqld
setting the start level - chkconfig --level 345 mysqld on
This example applies to any server and to validate that it worked you can use chkconfig --list | grep mysql to see the changes
I had been looking for a simple way to get a thread dump from JBoss to see what was happening on each of the SpatialKey servers without actually logging onto any of them. After all reading through thread dumps is one of my favorite past times. I found a simple way to do that writes an HTML file containing the thread dump and I can access it by hitting the webserver directory at a hidden URL.
The command is rather simple to generate a thread dump:
I found a helpful command today to help me search through many different file in linux. I have a bunch of log files and want to find a certain occurrence of an error.
for i in `find ./`; do grep -H "string to search" $i; done
Excel is a powerfull tool for sorting and analyzing data but what if you want to look at your data from another angle. What if you want to see the geographic aspect and understand the trends both over time and geographically? Geocoding your data can be a challenging task and getting it on a map can be even more of a pain. SpatialKey takes the pain out of mapping your excel data and makes it simple to get your data into a map and build engaging reports.
With SpatialKey we create AMI's that are exact replicas of each other and can be scaled easily all of the persistent content is stored in a EBS volume allowing us to deploy a new instance from a Snapshot and easy backup with Snapshots.
We recently moved our website for SpatialKey onto Amazon Ec2 which has worked great but there has been one items that has bugged me. If I needed to make a few small changes to the site I was using vi to make the inline edits. For larger changes I have scripts to scp the files up to the instance but the workflow for changes has become a pain.
I stumbled upon a great tool called ExpanDrive that allows me to use use sftp, basically ssh to manage my files remotely but leverage my Mac tools to do the editing. ExpanDrive provides a 30 day free trial but it is worth the price. The package runs $39.95. You can find answers to nearly all of your support questions on getsatisfaction.com. Right after installing I ran into an issue and found the answer right away.
If you are using Ec2 then you will probably experience this issue as well. Since Ec2 doesn't use a username/password combo but instead uses a cert/keypair for authentication. You need to use ssh-add to add the keypair then you can use it in ExpanDrive.
To do this open up a terminal session and run the following command - "ssh-add /Users/myusername/myec2keys/id_my_keypair"
Next I set up ExpanDrive with an empty password and it logged right in and mounted the drive on my Mac. I could then open up any of my Mac editing tools to edit on the server.
It has been a long time in the works but we are finally in a private beta for SpatialKey. This version has many more features and capabilities beyond our technology preview that was released in August of 2008 and provide a slick new ability to upload your own data.
Instead of me rambling on about what it does and doesn't do just check out the video!
Apply for the private beta if you are intersted in visualizing your own data.
Utah State Senator and Ogden City Police Chief blogs about Spatialkey and the power of bringing together relevant data sets from multiple places. Here is a small sampling of Chief Griener's posting.
"Not just any map - it's a combination map of satellite images, street map, and my geographically assigned patrol beat map in layers. The company (Universal Mind) has made it so user friendly that I can literally research and plot over 400,000 calls for service in the last 5 years in about 30 seconds. The system is web based so quickly analyzing What-Ifs can be accomplished anytime, anywhere, on any computer with an air card. You can begin to see how this would be valuable to me and my law enforcement team."
SpatialKey Law Enforcement continues to evolve and improve to provide cutting edge technology to the Law Enforcement community. It is exciting to be a part of great team with so much potential. Keep an eye on the SpatialKey blog for other important news around SpatialKey in general.
The most recent issue of ComputerWorld magazine features a cover story called "Can Web 2.0 Save B.I.?" that features a case study on SpatialKey. In the article they interview Chief Jon Greiner of the Ogden Police Department in Utah. Ogden is the first installation of the enterprise version of SpatialKey Law Enforcement Dashboard (see the press release), and Universal Mind has been working closely with the Ogden PD to use the SpatialKey platform to develop what we think is a game-changing crime mapping product.
If you have been working with computers very long you have probably experienced that sinking feeling when you realize that you have a dead hard drive without a recent backup of your important files. Well this happened to me this weekend with my blog and I did have a backup but it was from April 2007. I can't explain how mad I was at myself because I had just thought about running a backup earlier in the week before the server failed. While I have not been blogging much recently I had posted several entries since April of last year and I couldn't think of a way to get them back. Then I realized that nearly every HTML page on the internet is cached by Google, it is amazing when you think about it. I started searching on google for specific blog entry IDs, here is an example "site:bpurcell.org 1101". This brought up the specific link, then I opened the cached version and saved each entry to disk. This not only included the content of the blog entry but also the comments. Luckily I have been storing my images on flickr recently so I did not have to go searching for my images. It took a while to pull together the entries but after a few hours I had it rebuilt. I still need to add the comments but at least I have the site back up and running, along with a better backup strategy for the future!
After leaving Adobe in November of 2007 I joined Universal Mind as the director of Technology. UM had been working on a product that was called LaunchPad. At that time LaunchPad was a proof of concept built for the San Francisco PD with basic heatmap capabilities and no server side architecture. Our goal was to pull together a team with GIS experience, data visualization and build a kick ass product based on LaunchPad. As a consulting organization it is a unique endeavor to go off and build a product that is not targeted toward any specific customer. With a small team and some part time resources we set out to build what is now known as SpatialKey. Over the last nine months I have had the pleasure of managing an amazing group of resources on the SpatialKey project. Initially we started with a very small team of Doug McCune and Ben Stucki. Anyone in the Flex community recognizes Doug and Ben as amazing Flex developers and it did not take me long to realize the same thing. I am amazed at the things they can do with Actionscript. In late December we hired Reggie Wilbanks. Reggie had worked in traditional GIS for a long time and brought us a lot of experience and quickly became a skilled AS3 and Flex developer. Andy Powell has also been a key resource while working with us part time in addition to customer facing projects. Zach Johnson joined our team in June straight out of University of Wisconsin with a masters in Cartography. Zach has great passion and skill for online mapping and data visualization and is a great writer as well. Behind the scenes Mike Connor has served as our product manager, Mike has been a great help to me in getting this product developed and shaping the Law Enforcement version. Tom Link has been the overall leader behind the project as Universal Mind's CTO. Thanks to Tom for bringing me to Universal Mind and giving me the great oppportunity to lead such an amazing team.
I am excited to finally introduce SpatialKey as a a next generation Information Visualization, Analysis and Reporting System. It is designed to help organizations quickly assess location based information critical to their organizational goals, decision making processes and reporting requirements. The technology preview allows you to use our visualization templates to look at sample datasets that we have collected over the last few months. A future release based on Adobe AIR will all users to import their own data and bring into the visualization templates to look at the data in multiple ways.
Why is it different?
Example view of SpatialKey
We've been seeing the same approach to web-based mapping for years now. It is pretty much all the same with just markers on a map. Little pin markers work fine if you're showing a few data points. If you are looking for the closest Star Bucks to your home you can pull up google, do a search and you see all of the Star Bucks within X miles of your home. Markers work great for that, but what if you wanted to see all of the StarBucks in the United States or play back the growth of Star Bucks as they grew from just a single coffee shop in Seattle to the thousands of locations around the country? Or only see the StarBucks built in the US during the 1990s? SpatialKey allows you to view thousands of points without the need of any server technology.
SpatialKey uses some of the most advanced visualization renderings for geospatial data that have ever been seen on the web. The focus here is on aggregate renderings: heatmaps, thematic grids, graduated circles. For large quantities of data you want to see density or sum total value. Piling a bunch of markers on top of each other ends up confusing users without a clear picture or story for the data. SpatialKey focuses on rendering aggregate data in meaningful ways. We can show a heatmap of the entire country or the entire world. and let you visualize any number of data fields. The technology preview is just using CSV files and loads them in at runtime, nothing is pre-rendered and you can intereact with the applications to filter the data and see updates within seconds. What you won't see (yet at least) is the same capability with a server implementation allowing you to do the same thing but with Millions of points. We are already doing this with our Law Enforcement version but could bring the same capability to other industries.
What can you do now?
Go check out our gallery pairing up our four visualization templates that launched with the Technology preview. We will be putting up videos in the near future to showcase the features and provide a walkthrough for each template. If you have a particular dataset that you think would be compelling to see use the contact us form to send us your information and we will be in touch to collect the data. Also sign up for the beta to be notified when the desktop application becomes available so that you can import and view your own data.
Something has really been annoying me with Firefox
3 where it would open a new tab for pop up windows. It was particular
annoying when I was using a Content Management System that normally
opens an HTML editor in a popup window and resize the popup window to a
particular size. Instead of opening a new window the HTML editor would
open in a new tab and resize the browser so that after I was done with
the HTML editor I had to resize the browser back to its original size.
I searched in the configuration for Firefox but the only configuration
change is to change the behavior so that new links open in a new window
and not a new tab, there was nothing about changing the behavior for
pop ups.
With a little searching around Google I found a solution that solves
the problem. The first step is to enter about:config into the address
window to enter the advanced configuration panel. In the filter enter
browser.link.open_newwindow.
The original setting for "browser.link.open_newwindow" is set to 3,
set this value to 2 close the browser and reopen it. Now Firefox will
open popup windows normally. I found the tip here.
Last year we purchased the Safety First Kidtrax
Corvette for my son from Toys R Us. After having it only a few months
the battery stopped holding a charge and it would run down after a few
minutes. I went to Toys R Us to find a replacement battery and they did
not have them, they couldn't even order them. ? I started looking on
the internet and found that the Safety First batterys are almost $90
with shipping which I found a bit ridiculous. I decided to try and find
an aftermarket battery and found one that is normally used in UPS's.
I will be presenting again at Max this year in San
Franscico on the topic of "Architecting ColdFusion For Scalability And
High Availability". I want to take a new approach this year and solicit
input on specific areas to cover during the presentation. In some cases
you come to Max year after year and hear the same thing and since I
have only 60 minutes and a lot of content in my head I would like to
gather input on what areas the community would find interesting. There
are two main areas where I can focus, coding practices for scalability
and clustering architectures for high availability. I think for the
most part developers attending Max understand best practices for coding
(although it doesn't mean they always follow them). The clustering
architectures and options have not changed much over the years since
CFMX 6 so many of my old articles still apply. I would be interested to
hear general thoughts on this topic and areas where I could focus the
presentation. My outline is due on the 14th but I think I will probably
tweak things all the way through August.
