bpurcell.org - Blog
Calendar
SunMonTueWedThuFriSat
            1
2 3 4 5 6 7 8
9 10 11 12 13 14 15
16 17 18 19 20 21 22
23 24 25 26 27 28 29
30            

Subject Archives
Amazon EC2 (15)
ColdFusionMX (155)
Corvette (3)
Flash Lite (4)
Flash Media Server (5)
Flash Player (3)
Flex (39)
General Web Dev (14)
HDTV (3)
Jboss (1)
Jquery (2)
JRun (59)
Max 2003 (3)
Other (33)
PC Hardware (16)
Software (17)
SpatialKey (7)
Wireless (8)
Working Out (1)

RSS Feed
Feed Listing

Site Contents
Home
My Blog
ColdFusion
JRun
Flex
Wireless & Networking
Hardware & Gadgets
HDTV
Software Picks
Pictures
Contact Me
Search


My Articles & Papers
Flex Performance
Server/Client
mm.com Under the Hood
Multiple Instances of CFMX
Multiple Instance Config
NLB with ColdFusion
Clustering CFMX for J2EE
Multi-Tier Hardware LB w CFMX
Cisco CSS & Coldfusion MX
JRun 4 Jini based Clustering
WiFi Growth

2ID Tacweb

Other Hobbys
Body-For-Life Challenge

Personal Projects
Family Pool
Deck Gate

username:
password:
 

 
March 30, 2010

I was running out of space on a server the other day and was wondering what directory was full and what files I could clean up. There is a simple command that is helpful to identify usage by directory. This works from within any directory so you can walk the path from / down through your largest directories and look for files to clean up.

[root@localhost]# du -hc --max-depth=1
4.0K ./misc
4.0K ./srv
576K ./root
29M ./sbin
16K ./tmp
2.0G ./usr
731M ./proc
12M ./boot
0 ./sys
379M ./appserver
28M ./opt
447M ./var
85M ./lib
8.0M ./etc
11M ./home
0 ./selinux
6.3M ./bin
16K ./lost+found
40K ./dev
24K ./mnt
3.7G .
3.7G total

January 6, 2010

We use PostgreSQL for SpatialKey and because of that I have learned a lot about system administration with PostgreSQL. While debugging an issue with tables that had ballooned up I ran into this article which has some great tips for isolating db and table size with a few quick commands. http://www.thegeekstuff.com/2009/05/15-advanced-postgresql-commands-with-examples/

Also check out 15 Practical PostgreSQL database administration commands and backup and restore PostGreSQL with pg dump and psql.

 

August 4, 2009

There is nothing more frustrating than taking 30 minutes filling out some text in a long form only to submit and have your session reset and lose all of your data. Here is a simple solution to keep a session open on forms where users may need an extended period of time for data entry. I included this code on any page where I need the session extended. You can remove the javascript include if JQuery is included on everyone of your pages already.

<!--- Include this file on any page where users may take a 
long time filling in text and you want to keep the session open --->
<cfparam name="variables.refreshrate" default="60000">

<cfoutput>
<script type="text/javascript" src="/js/jquery-1.3.2.min.js"></script>
<script language="JavaScript" type="text/javascript">
	$(document).ready(function(){
 					setTimeout("callserver()",#variables.refreshrate#); 
 				});
 	
 	function callserver()
 	{
 		var remoteURL = '/emptypage.cfm	';
			    $.get(remoteURL,  function(data){
			      //alert(data);
			      setTimeout("callserver()",#variables.refreshrate#);
			    });
 	}
</script>
</cfoutput>

July 30, 2009

James Ward and Jon Rose recently published the latest episode of Drunk on Software with members of the SpatialKey team, including myself, Tom Link, and Doug McCune.


In the video we talk about Universal Mind, how SpatialKey came to be, a lot about what SpatialKey is and how it works. I dive into the architecture a bit and how we leveraged the cloud to build out our infrastructure.

Thanks to Jon and James hosting us, we crashed Jon's house for the video. I think Doug tossed back the most beers I can't keep up with him and still talk.

June 11, 2009

This issue drove me nuts today for several hours today, to be honest I was pretty close to breaking something! Earlier in the day I had the NFS mount working fine then I created an AMI and booted up another instance from the newly created AMI but in the new instance the mount kept failing. The error looks something like this:

[root@server]# mount -t nfs 192.168.2.1:/dbshare /mnt/dbshare
mount: 192.168.2.1:/dbshare failed, reason given by server: Permission denied

According to the error you would think that I have a configuration issue so I changed everything that I could think of within /etc/exports

My /etc/exports originally looked like this:

/dbshare 192.168.2.2(rw,sync) (where 192.168.2.2 is the client where I am performing the mount)

I changed it to something more open like this with no luck:

/dbshare 192.168.0.0/255.255.0.0(rw,sync)

I started looking around the logs on the server in /var/log/messages and found that it was authenticating fine
Jun 11 19:04:00 servername mountd[5222]: authenticated mount request from 192.168.2.2:736 for /dbshare (/dbshare)

I was really frustrated at this point and I had already spent an hour on Google looking for the answer. I found another answer but the website was down, luckily the cached version on Google came to the rescue.

The ANSWER:

The problem was that the special nfsd file system that mounts to /proc/fs/nfsd wasn't mounted. I'm not sure how it gets mounted (maybe rc.sysinit does it?), but I tool the advice from the forum entry and added an entry to /etc/fstab

none /proc/fs/nfsd nfsd auto,defaults 0 0

then ran mount -a

After this the mount worked fine. I hope that someone finds this helpful.

May 27, 2009

I often need to copy code from a SVN repository to a location where I have files deployed or I may pull code from an SVN repository on the web and want to check it in locally. Both of these actions require the stripping out of all .svn directories spread throughout the directory structure. On Linux or Mac this is pretty simple with the following command

sudo find -d . -name '.svn' -exec rm -rf '{}' \; -print

May 14, 2009

I ran across this today. Examples are great when trying to understand JQuery selectors.

In recent months I have been using PHP to extend functionality inside of WordPress for the SpatialKey site and working with JQuery as well. After working with ColdFusion for so long I have gotten spoiled by the CFDUMP tag . Here are a few options similar to CFDUMP for both PHP and Javascript.

CFDUMP for PHP - dBug

CFDUMP for Javascript

April 29, 2009

One challenge I have encountered with Amazon Ec2 is the sending of email from our web applications. If you try to send directly from sendmail or postfix then you might as well forget about guranteed delivery. A large amount of your email will end up in spam folders if it is even delivered at all.

There are a few problems with delivering email from Ec2

  • MX record will not map to your IP and you are using dynamic IPs (you can address this with elastic IP's) adding a SPF DNS record can help as well
  • Reverse DNS will map back to Amazon and not back to your Hostname
  • Many of the Ec2 IP's have been blacklisted due to abusers of the service sending spam.

There are a few solutions to this problem and I will propose two of them.

Using a google apps account:

If you are using a solution like Google Apps and have them host your email accounts then you can use gmail as your SMTP server. You will need to create an account donotreply@mydomain.com and use authentication in your applications to send the email. With Google apps you cannot override the "from" address when you send email it wil always become whatever you account you are sending from. For example if you create the account donotreply@mydomain.com and attempt to set the "from" in your code to send from brandon@mydomain.com google will override it and send from donotreply. The only option is to set "replyto" in your code and when a user replys they will send to your replyto account. With google apps you are limited to a maximum amount of 500 emails a day per account as well and if you are sending a lot of emails this can quickly become a problem. This is a great solution for small volumes of email and you delivery rates are very good.

Relay from localhost through a third party:

This blog post outlines a set of steps to relay through a local Postfix instance to a third party SMTP service. The great thing about this solution is that you can send email from your application to localhost without storing the authentication parameters in your applications code and have Postfix handle it all. If you have many applications sending email this can greatly simplify things. Also it allows your application to hand off the emails quickly to another service that can handle queuing in case the third party email service is down at any time. You could combine the approach above with this one but you would still have the 500 email limitation. I am searching for a good third party SMTP service that is reliable, the author of the blog post recommends AuthSMP. I have not tried them and their prices are not too high but not cheap either. I am going to do a little more digging and test some of the options and will report back to this posting.

This is common knowledge if you have been using Linux for a while but I still find it a helpful resource to understand how you set what programs are running when Linux starts. This is mainly specific to Red Hat or CentOS which I use on a regular basis.

Running level
Running level is the current running functional level of the operation system, from level 1 to 6, possessing different functions.
Here are the different running levels:

  • 0 - downtime or halt
  • 1 – single user mode
  • 2 – multi-user,no NFS
  • 3 – entire multi-user mode(standard running level)
  • 4 – not available
  • 5 - X11 multi-user graphic mode(xwindow)
  • 6 - restart (never set initdefault 6 )

These levels are specified in the file /etc/inittab., which is the main file that the program init looks for, and the first running service is placed under the directory /etc/rc.d. For most Linux releases, startup scripts are all located in /etc/rc.d/init.d, which are all linked to the directory /etc/rc.d/rcn.d by ln command (here the n is the running level 0-6).

Setting services/applications to run at startup using chkconfig

chkconfig command (under redhat and centos)

Linux provides the command chkconfig to update and query system services of different running levels allowing you to set when certain process are started.

Syntax:
chkconfig --list [name]
chkconfig --add name
chkconfig --del name
chkconfig [--level levels] name
chkconfig [--level levels] name

example:

[user@server]# chkconfig --list
anacron        	0:off	1:off	2:on	3:on	4:off	5:on	6:off
atd            	0:off	1:off	2:off	3:on	4:on	5:on	6:off
autofs         	0:off	1:off	2:off	3:on	4:off	5:on	6:off
conman         	0:off	1:off	2:off	3:off	4:off	5:off	6:off
crond          	0:off	1:off	2:on	3:on	4:on	5:on	6:off
dc_client      	0:off	1:off	2:off	3:off	4:off	5:off	6:off
dc_server      	0:off	1:off	2:off	3:off	4:off	5:off	6:off
getsshkey      	0:off	1:off	2:off	3:off	4:on	5:off	6:off
haldaemon      	0:off	1:off	2:off	3:on	4:on	5:on	6:off
haproxy        	0:off	1:off	2:off	3:on	4:on	5:on	6:off
httpd          	0:off	1:off	2:off	3:off	4:off	5:off	6:off
ibmasm         	0:off	1:off	2:off	3:off	4:off	5:off	6:off
ip6tables      	0:off	1:off	2:on	3:on	4:on	5:on	6:off
iptables       	0:off	1:off	2:on	3:on	4:on	5:on	6:off
jexec          	0:on	1:on	2:on	3:on	4:on	5:on	6:on

chkconfig has five functions: add service, delete service, list service, change startup info and check the start state of specified service.

Option overview:

--level levels
specify running level, which is a character string composed of the number 0 to 7. For example:
--level 35 means to specify running level 3 and 5.
To stop the service nfs during running level 3,4,5, use the command next: chkconfig --level 345 nfs off

--add name
This option adds a new service, chkconfig ensures every running level an entrance to start (S) or to kill (K). if it is absent, then it would auto establish from default init script.

--del name
To delete service and delete related sign connections from /etc/rc[0-6].d.

--list name
List, if name is specified, then it only displays specified service name, otherwise, to list the state of all service at different running levels.

Usage examples:

As an example if you wanted mysql to run when the os starts you just need to do the following

  • /etc/init.d/mysqld must exist and needs to be an executable (chmod +x)
  • Add mysql - chkconfig --add mysqld
  • setting the start level - chkconfig --level 345 mysqld on

This example applies to any server and to validate that it worked you can use chkconfig --list | grep mysql to see the changes

 

I ran across this article while debugging some .htaccess issues on our SpatialKey server and found some helpful information on .htaccess.

April 23, 2009

This is a good article on some monitoring tools for Linux. I had already used several of them before but found 'htop' and 'atop' in the comments and they are great

I had been looking for a simple way to get a thread dump from JBoss to see what was happening on each of the SpatialKey servers without actually logging onto any of them. After all reading through thread dumps is one of my favorite past times. I found a simple way to do that writes an HTML file containing the thread dump and I can access it by hitting the webserver directory at a hidden URL.

The command is rather simple to generate a thread dump:

/jboss/bin/twiddle.sh invoke "jboss.system:type=ServerInfo" listThreadDump > /dump.html

I modified the /etc/crontab file to create this file every minute

*/1 * * * * root //jboss/bin/twiddle.sh invoke "jboss.system:type=ServerInfo" listThreadDump > /apachewebroot/subdir/dump.html 2>&1

It is pretty simple but it worked for what I needed.

April 7, 2009

I found a helpful command today to help me search through many different file in linux. I have a bunch of log files and want to find a certain occurrence of an error.

for i in `find ./`; do grep -H "string to search" $i; done

March 27, 2009

Excel is a powerfull tool for sorting and analyzing data but what if you want to look at your data from another angle. What if you want to see the geographic aspect and understand the trends both over time and geographically? Geocoding your data can be a challenging task and getting it on a map can be even more of a pain. SpatialKey takes the pain out of mapping your excel data and makes it simple to get your data into a map and build engaging reports.

Spreadsheet_to_report_Spatialkey.jpg

I have outlined this process in an article explaining how to take your excel data and get it into SpatialKey

March 25, 2009

With SpatialKey we create AMI's that are exact replicas of each other and can be scaled easily all of the persistent content is stored in a EBS volume allowing us to deploy a new instance from a Snapshot and easy backup with Snapshots.

This article is the best one I have seen highlighting performance with EBS volumes. He also highlights using software based RAID and the performance numbers with RAID are impressive. I am loving Amazon Ec2 more and more each day.

We recently moved our website for SpatialKey onto Amazon Ec2 which has worked great but there has been one items that has bugged me. If I needed to make a few small changes to the site I was using vi to make the inline edits. For larger changes I have scripts to scp the files up to the instance but the workflow for changes has become a pain.

I stumbled upon a great tool called ExpanDrive that allows me to use use sftp, basically ssh to manage my files remotely but leverage my Mac tools to do the editing. ExpanDrive provides a 30 day free trial but it is worth the price. The package runs $39.95. You can find answers to nearly all of your support questions on getsatisfaction.com. Right after installing I ran into an issue and found the answer right away.

If you are using Ec2 then you will probably experience this issue as well. Since Ec2 doesn't use a username/password combo but instead uses a cert/keypair for authentication. You need to use ssh-add to add the keypair then you can use it in ExpanDrive.

To do this open up a terminal session and run the following command - "ssh-add /Users/myusername/myec2keys/id_my_keypair"

Next I set up ExpanDrive with an empty password and it logged right in and mounted the drive on my Mac. I could then open up any of my Mac editing tools to edit on the server.

It has been a long time in the works but we are finally in a private beta for SpatialKey. This version has many more features and capabilities beyond our technology preview that was released in August of 2008 and provide a slick new ability to upload your own data.

Instead of me rambling on about what it does and doesn't do just check out the video!

Apply for the private beta if you are intersted in visualizing your own data.

September 8, 2008

Utah State Senator and Ogden City Police Chief blogs about Spatialkey and the power of bringing together relevant data sets from multiple places. Here is a small sampling of Chief Griener's posting.

"Not just any map - it's a combination map of satellite images, street map, and my geographically assigned patrol beat map in layers. The company (Universal Mind) has made it so user friendly that I can literally research and plot over 400,000 calls for service in the last 5 years in about 30 seconds. The system is web based so quickly analyzing What-Ifs can be accomplished anytime, anywhere, on any computer with an air card. You can begin to see how this would be valuable to me and my law enforcement team."

SpatialKey Law Enforcement continues to evolve and improve to provide cutting edge technology to the Law Enforcement community. It is exciting to be a part of great team with so much potential. Keep an eye on the SpatialKey blog for other important news around SpatialKey in general.

SpatialKey_ComputerWorld.jpg

The most recent issue of ComputerWorld magazine features a cover story called "Can Web 2.0 Save B.I.?" that features a case study on SpatialKey. In the article they interview Chief Jon Greiner of the Ogden Police Department in Utah. Ogden is the first installation of the enterprise version of SpatialKey Law Enforcement Dashboard (see the press release), and Universal Mind has been working closely with the Ogden PD to use the SpatialKey platform to develop what we think is a game-changing crime mapping product.

Yes that is our application on the cover. Check out the online version of the article on ComputerWorlds' website

September 2, 2008

If you have been working with computers very long you have probably experienced that sinking feeling when you realize that you have a dead hard drive without a recent backup of your important files. Well this happened to me this weekend with my blog and I did have a backup but it was from April 2007. I can't explain how mad I was at myself because I had just thought about running a backup earlier in the week before the server failed. While I have not been blogging much recently I had posted several entries since April of last year and I couldn't think of a way to get them back. Then I realized that nearly every HTML page on the internet is cached by Google, it is amazing when you think about it. I started searching on google for specific blog entry IDs, here is an example "site:bpurcell.org 1101". This brought up the specific link, then I opened the cached version and saved each entry to disk. This not only included the content of the blog entry but also the comments. Luckily I have been storing my images on flickr recently so I did not have to go searching for my images. It took a while to pull together the entries but after a few hours I had it rebuilt. I still need to add the comments but at least I have the site back up and running, along with a better backup strategy for the future!

September 1, 2008

After leaving Adobe in November of 2007 I joined Universal Mind as the director of Technology. UM had been working on a product that was called LaunchPad. At that time LaunchPad was a proof of concept built for the San Francisco PD with basic heatmap capabilities and no server side architecture. Our goal was to pull together a team with GIS experience, data visualization and build a kick ass product based on LaunchPad. As a consulting organization it is a unique endeavor to go off and build a product that is not targeted toward any specific customer. With a small team and some part time resources we set out to build what is now known as SpatialKey. Over the last nine months I have had the pleasure of managing an amazing group of resources on the SpatialKey project. Initially we started with a very small team of Doug McCune and Ben Stucki. Anyone in the Flex community recognizes Doug and Ben as amazing Flex developers and it did not take me long to realize the same thing. I am amazed at the things they can do with Actionscript. In late December we hired Reggie Wilbanks. Reggie had worked in traditional GIS for a long time and brought us a lot of experience and quickly became a skilled AS3 and Flex developer. Andy Powell has also been a key resource while working with us part time in addition to customer facing projects. Zach Johnson joined our team in June straight out of University of Wisconsin with a masters in Cartography. Zach has great passion and skill for online mapping and data visualization and is a great writer as well. Behind the scenes Mike Connor has served as our product manager, Mike has been a great help to me in getting this product developed and shaping the Law Enforcement version. Tom Link has been the overall leader behind the project as Universal Mind's CTO. Thanks to Tom for bringing me to Universal Mind and giving me the great oppportunity to lead such an amazing team.

I am excited to finally introduce SpatialKey as a a next generation Information Visualization, Analysis and Reporting System. It is designed to help organizations quickly assess location based information critical to their organizational goals, decision making processes and reporting requirements. The technology preview allows you to use our visualization templates to look at sample datasets that we have collected over the last few months. A future release based on Adobe AIR will all users to import their own data and bring into the visualization templates to look at the data in multiple ways.

Why is it different?


Example view of SpatialKey

We've been seeing the same approach to web-based mapping for years now. It is pretty much all the same with just markers on a map. Little pin markers work fine if you're showing a few data points. If you are looking for the closest Star Bucks to your home you can pull up google, do a search and you see all of the Star Bucks within X miles of your home. Markers work great for that, but what if you wanted to see all of the StarBucks in the United States or play back the growth of Star Bucks as they grew from just a single coffee shop in Seattle to the thousands of locations around the country? Or only see the StarBucks built in the US during the 1990s? SpatialKey allows you to view thousands of points without the need of any server technology.

SpatialKey uses some of the most advanced visualization renderings for geospatial data that have ever been seen on the web. The focus here is on aggregate renderings: heatmaps, thematic grids, graduated circles. For large quantities of data you want to see density or sum total value. Piling a bunch of markers on top of each other ends up confusing users without a clear picture or story for the data. SpatialKey focuses on rendering aggregate data in meaningful ways. We can show a heatmap of the entire country or the entire world. and let you visualize any number of data fields. The technology preview is just using CSV files and loads them in at runtime, nothing is pre-rendered and you can intereact with the applications to filter the data and see updates within seconds. What you won't see (yet at least) is the same capability with a server implementation allowing you to do the same thing but with Millions of points. We are already doing this with our Law Enforcement version but could bring the same capability to other industries.

What can you do now?

Go check out our gallery pairing up our four visualization templates that launched with the Technology preview. We will be putting up videos in the near future to showcase the features and provide a walkthrough for each template. If you have a particular dataset that you think would be compelling to see use the contact us form to send us your information and we will be in touch to collect the data. Also sign up for the beta to be notified when the desktop application becomes available so that you can import and view your own data.

Something has really been annoying me with Firefox 3 where it would open a new tab for pop up windows. It was particular annoying when I was using a Content Management System that normally opens an HTML editor in a popup window and resize the popup window to a particular size. Instead of opening a new window the HTML editor would open in a new tab and resize the browser so that after I was done with the HTML editor I had to resize the browser back to its original size. I searched in the configuration for Firefox but the only configuration change is to change the behavior so that new links open in a new window and not a new tab, there was nothing about changing the behavior for pop ups.

With a little searching around Google I found a solution that solves the problem. The first step is to enter about:config into the address window to enter the advanced configuration panel. In the filter enter browser.link.open_newwindow.

The original setting for "browser.link.open_newwindow" is set to 3, set this value to 2 close the browser and reopen it. Now Firefox will open popup windows normally. I found the tip here.

Last year we purchased the Safety First Kidtrax Corvette for my son from Toys R Us. After having it only a few months the battery stopped holding a charge and it would run down after a few minutes. I went to Toys R Us to find a replacement battery and they did not have them, they couldn't even order them. ? I started looking on the internet and found that the Safety First batterys are almost $90 with shipping which I found a bit ridiculous. I decided to try and find an aftermarket battery and found one that is normally used in UPS's.

Here is the set of steps and the battery to get you up and running

I will be presenting again at Max this year in San Franscico on the topic of "Architecting ColdFusion For Scalability And High Availability". I want to take a new approach this year and solicit input on specific areas to cover during the presentation. In some cases you come to Max year after year and hear the same thing and since I have only 60 minutes and a lot of content in my head I would like to gather input on what areas the community would find interesting. There are two main areas where I can focus, coding practices for scalability and clustering architectures for high availability. I think for the most part developers attending Max understand best practices for coding (although it doesn't mean they always follow them). The clustering architectures and options have not changed much over the years since CFMX 6 so many of my old articles still apply. I would be interested to hear general thoughts on this topic and areas where I could focus the presentation. My outline is due on the 14th but I think I will probably tweak things all the way through August.

 
Page Render Time:312