Traffic Metrics Require Context

Oct 2 2012

Last Friday, Chris posed a question to the team regarding the traffic numbers that various analytics tools provide: why do we trust the numbers so much, regardless of their source? This got me thinking… why do we put so much faith into the numbers we’re looking at? Are we applying the correct context to these metrics?

I did some further research into two different tools we leverage at RD2, Inc. (AWstats and Google Analytics) to provide a reason as to why the numbers don’t line up.

AWstats (Awwww, stats!)

AWstats is a product of the late 90s as a solution to address the concerns of “how many people are visiting my website?”; I can only assume web hosting during that time wasn’t nearly as cheap as it is today and ROI was an even bigger focus than it is now. Getting people to the website then was a big deal; today, getting traffic could be viewed as a relatively easier task (given the social media technologies at our disposal.) AWstats has been updated several times over the years, but its base function has remained the same: present the data in access log files in a way that makes sense to the layman (read: graphs and tables.) This is where the question, “So it’s like Google Analytics?” comes up and my response is along the lines of, “Ehh… in a way, but not really.” AWstats provides a more easily read representation of the server access logs… so what does that mean when talking “traffic” and “visits” and “page views”? Well to be quite honest, it means nothing without an understanding of how the tool works and appropriate context is applied to what is being viewed (we’ll get to this in a bit.)

Google Analytics (or, cloud-computing crunching the numbers and giving you insane amounts of detail on your visitors behavior)

Positioned neatly among the many tools found in the Google webmaster’s “tool belt”, Google Analytics is a platform offering many different methods of quantifying, displaying, processing, comparing, and filtering data on user behavior while visiting a web property. Google doesn’t charge for their service (which is interesting, seeing as it’s one of the more powerful tools they offer) so pretty much anyone with a Google ID can use it. When discussing the “how” tracking occurs, Google’s method is completely different: JavaScript.

So why would I use both?

Scenario: Company X provides a variety of consumer services online and in-person. After launching a new version of their social home base, Company X wants to keep a diligent eye on visitor traffic patterns as a means of quantifying whether or not the new platform is successful. Eager Analyst 1 working for Company X goes to Google Analytics and prepares a detailed report based on this information; Eager Analyst 2 prepares a similar report to Eager Analyst 1′s Google Analytics report, but opts to include data from AWstats. During the presentation, all of the numbers line up between the two reports until AWstats is discussed. For the first quarter of the year, Google Analytics reports 613,718 total pages viewed, whereas AWstats reports 3,740,453 total pages viewed. Logic tells us that these 2 numbers don’t really match up; AWstats is reporting approximately 6.1 times higher than Google Analytics. So which number is correct? As it turns out, both are. Stay with me for a minute and I’ll explain why that is… If you recall, I mentioned a tool is only valuable when the correct context is applied; in this case, it’s the definitions of the terms being used. This is key when reviewing information from any source, e.g., store foot traffic metrics, the number of customers served, and the number of visitors. So we know that Google Analytics tracks visitors using JavaScript upon page load and AWstats reads log files, great; is it possible Google has a margin of error this large? Highly unlikely… Digging into the definitions used for “page” with both services, I was surprised to find that the definitions are widely different. Google Analytics defines a page view as a single page (rd2inc.com and rd2inc.com/team are 2 different pages to Google Analytics, loading both pages twice would result in 4 page views); AWstats definition is very different. When calculating “page” views, AWstats calculates the page itself, any additional PHP functions run to display dynamic content, and AJAX queries to the server that occur after the page has loaded (so for each 1 page, there could be 6+ page “views” when a content management system is used for page generation.)

Aside from pages viewed, another variable metric differing between the two is how user sessions are tracked. By default, AWstats tracks user sessions in what is roughly 60 min segments and Google Analytics tracks in roughly 30 minute segments. Since Google Analytics is JavaScript based, multiple users from the same public IP address are tracked separately; AWstats would view all visitors from the same public IP address as one unique visitor, and if multiple people from the same public IP address visit Company X’s website within approximately 60 minutes of the first visit from said IP address, the number of unique visitors will be greatly skewed (this is a limitation stemming from how AWstats captures its data.)

The take away from this shouldn’t be that AWstats is antiquated and shouldn’t be used; it should be understood that each tool has its strengths and opportunities (AWstats can track which pages are being crawled by search engines, Google Analytics can’t.)

So go forth, review the traffic metrics and make sure the data being presented is presented with the correct context applied

Reference URLs
AWstats

Tags

, , , , , ,

Rusty Smith On RD2

Aug 30 2012

We have to think like we are a part of each others’ companies in order to innovate. With so many moving parts and user scenarios, it takes a tightly integrated set of teams and tremendous focus. Rusty Smith and the team at Tyler Technologies have become great partners through the years. We’re happy to see this new project rolling out. Look for more from Tyler Technologies on the CAD application in the near future!

Thanks Rusty for these great words. It’s been amazing to work with you and the team at Tyler.

Reference:

Tags

Tyler Tech Mobile CAD App

Aug 30 2012

We’ve been lucky enough to have worked with Tyler Technologies for a few years now. What does Tyler Technologies do? Here’s what it says on their website:

Tyler Technologies is the largest company in the United States exclusively focused on providing integrated software and technology services to the public sector. We deliver an expansive portfolio of software solutions that span the breadth and depth of the mission-critical services for the public sector.

  • Appraisal & Tax
  • Courts & Justice
  • ERP / Financial
  • Land & Vital Records
  • School

So, what does all this mean? Tyler has some great relationships with local governments and they serve them by helping to make their jobs more effective by bringing them innovative solutions. In fact, Tyler has been doing some amazing things and has been fearless with respect to bringing emerging tech to schools, courtrooms, police departments, and many more public scenarios. This one in particular, Mobile CAD, is pretty incredible.

The Tyler Technology team in Lubbock, Texas had an idea. In listening to the needs of their Customers, Police Departments, they constructed a strategy and a team around the concept of helping Police Officers and other first responders to experience improved communications and access to data from the field. To date, many police cars are equipped with a rugged laptop that links to the department dispatching system and also gives access to various records while the Police Officers are in the field. Those machines are a dated technology, are expensive to replace, and don’t represent the best form factors that are available today in technology. The nature of Police work suggests, obviously, that Officers depend on data and communications in ways that can at times be mission or life-critical. Improving the way Officers interact with that data may have both immediate and long term advantages to police forces who are trying to manage their day to day workflows and communications.

As part of this assignment, we at RD2 were invited to fly to a Police Department in Oklahoma to ride with Officers during a night shift. There were five of us in total (between RD2 and Tyler Tech) and we split up with our list of questions. Our goal was to observe and learn how these Officers do their job and how they depend on technology at almost all times. We patrolled nearly all night on the streets and came away with a lot of information that became the basis for our User Experience Strategy.

While the assignment was obviously focused on gathering data for the purposes of building this application, it seemed we all came away with a new respect for the work that Police Officers do. How many of us dream as a child of being a Police Officer? If asked the question in a classroom, I’d certainly be raising my hand. It’s a role we admire and a role we put on a pedestal in most situations. We depend on these people for so many things and they do a very difficult and dangerous job. These Officers often put their lives on the line for people they may have never met.

To this moment I can recall the first traffic citation that my officer issued. It was dark and I recall sitting in the car (we’re not supposed to leave the car) while she pulled over a vehicle. She was behind the vehicle and her spotlight was appropriately pointed at the backside of the car. When she got out and approached the car I got this sick feeling in my stomach. I recall observing this vulnerability. Even with the spotlight and the headlights of the car it was apparent you still could not see everything. Granted, this was a routine stop, and it all turned out OK, but it was obvious to me that those steps the Officer took on approaching the vehicle were potentially very dangerous ones. I just can’t imagine the more complex calls and the chances these officers take every day on their shifts.

So, obviously, I never became a Police Officer. But it did give us a great honor to have worked with a company such as Tyler Technologies to create an application that was designed to help these public servants. These are human beings who are someones son or daughter. They are brothers and sisters. Many of them are mothers and fathers. They need our support and they need any tools that will give them additional safety. They need and deserve more applications like this that are forward thinking and that utilize the greatest things that technology has to offer to improve the nature of the jobs they do.

The following are actual screenshots of the iPad application that we worked with Tyler Technologies to create. As of recently, the city of Bartow, Florida is the first beta tester. Great work to the team and to the Tyler Tech Public Safety group in Lubbock to pull this all together. Stay on the lookout for more amazing innovation to come from this team!

  • Tyler Technologies Mobile CAD Press Release here
  • Public Safety iPad App website here

Tags

Grounded for Enhancements

Aug 24 2012

When you’re Southwest Airlines and you’re running one of the most recognized and awarded corporate blogs, you take your website performance and updates seriously. As the main portal for news and culture, Southwest wanted to make sure their users were getting the best experience from their website. For RD2, this meant not only taking a microscopic look at the website and it’s servers, but also tweaking the design and layout functionality to improve the websites usability.

On the front-facing side of the website, RD2 added new endless scroll functionality to the website so that users don’t have to click to get to more content, updated and re-engineered the log in process to make it smoother and easier to connect your Blog Southwest account to other accounts like , and upgraded the social streams to give users new ways to connect and engage.

Behind the curtain, RD2 worked heavily to test and optimize the website to deliver content to the user faster than ever before. While a site like this runs on a custom-built server, RD2 took things several steps farther, utilizing multiple rounds of content compression, virtual and real-time caches, and a host of server level customization tweaks that now allow the website to support a new level of concurrent users. When load testing the SWA Employee Blog with similar server configurations, it was able to support enough virtual traffic to warrant a direct call from the hosting company to complain about an unprecedented slowdown across their entire network.

Below is a full list of the enhancements.

Note: As part of the new RD2 initiative to create our new framework, we are integrating many of the tips, tweaks, and magic spells that were learned into our standard operating procedure.

Enhancements:

  • Implemented infinite scrolling on blog roll & comments
  • Turn off anonymous commenting
  • Updated and styled Facebook Connect
  • LUV Mail & SWA Star of the Month (read the LUV Mail post)
  • Commenting: Set order Newest to Oldest
  • Implemented “Featured Comments” and “LUV it” on Comments
  • Add Foodspotting, Instagram, and Pinterest to the “More Ways To Go Nuts” global feature
  • Removed Gowallas links and/or references from the “More Ways To Go Nuts” global feature and from within the user profile feature “Connect with Me”
  • Reduce Social Stream Spamming
  • Add Google+ to sharing
Tags

, , ,

Migrating from ExpressionEngine to WordPress using an XML / WXR import file

Aug 23 2012

One of my first projects here at RD2 was to migrate a site from ExpressionEngine over to WordPress. Part of that process involved making a hard decision:

“Do we migrate the content manually, create a custom database migration script, or is there a better way?”

It turned out that there was a much better way to accomplish what we were after. It was a little custom, but more importantly – it was reusable. The solution I came up with was to create a template within ExpressionEngine, that would output a WordPress XML / WXR import file. That sounds twisted, right?

Because we believe in open source and that it would truly be a disservice to the community of users and developers to keep this sort of thing to ourselves, we’ve decided to release this code snippet for everyone to use. Now thinking about migrating from ExpressionEngine to WordPress might not be such a challenge. You can tweak this to serve your own specific needs; I’ve set it up in a way that should cover enough ground for many use-cases.

Step 1

Create a new template in ExpressionEngine, put it in the root of your site, name it something like “my-export” for the URL.

Step 2

Set the “Type” to “XML”, mark the template to “Allow PHP”, allow “PHP Parse on Output”, and restrict the role to “Admin” so not just anyone can grab your data.

Step 3

Place the following code in the template and save.

EE->db;
echo '<' . '?' . 'xml version="1.0" encoding="UTF-8"' . '?' . '>';
?>




















 {exp:xml_encode}{site_name}{/exp:xml_encode}
 {site_url}
 Site
 2012-03-22T00:00:00 +0000
 en
 1.0
 {site_url}
 {site_url}
 http://expressionengine.com/

{exp:channel:entries channel="{segment_2}" orderby="date" sort="desc" status="open|featured|home_news" limit="9999" dynamic_start="yes" cache="no" refresh="0" disable="member_data|categories|category_fields|pagination"}
 
 {exp:xml_encode}{title}{/exp:xml_encode}
 {title_permalink}
 {entry_date format='%Y-%m-%d %h:%i:%s'}
 {exp:xml_encode}{author}{/exp:xml_encode}
 
 
 
 1000{entry_id}
 {entry_date format='%Y-%m-%d %h:%i:%s'}
 {gmt_entry_date format='%Y-%m-%d %h:%i:%s'}
 open
 open
 {exp:xml_encode}{url_title}{/exp:xml_encode}
 publish
 0
 0
 {segment_3}
 
 0
 

query( "SELECT
 exp_channel_fields.field_id,
 exp_channel_fields.field_name,
 exp_channel_fields.field_type
 FROM exp_channels
 LEFT JOIN exp_channel_fields
 ON exp_channel_fields.group_id = exp_channels.field_group
 WHERE exp_channels.channel_id = {channel_id}
 AND exp_channels.site_id = {entry_site_id}
 ORDER BY exp_channel_fields.field_order" );
 $content = '';
 $attachment = '';
 if ( 0 < $query->num_rows ) {
 foreach( $query->result_array() as $row ) {
 $field = $DB->query( "SELECT field_id_" . $row[ 'field_id' ] . " AS field_value FROM exp_channel_data WHERE entry_id = {entry_id} AND site_id = {entry_site_id} AND channel_id = {channel_id}" );
 if ( 0 == $field->num_rows )
 continue;
 $field = current( $field->result_array() );
 $field_value = $field[ 'field_value' ];
 $field_value = str_replace( '{' . 'filedir_3' . '}', 'http://' . $_SERVER[ 'HTTP_HOST' ] . '/images/backgrounds/', $field_value);
 if ( '{segment_4}' == $row[ 'field_name' ] )
 $content = $field_value;
 if ( '{segment_5}' == $row[ 'field_name' ] )
 $attachment = $field_value;
?>
 
 
 ]]>
 

 
 _thumbnail_id
 
 
 
{!--
EE->TMPL->modules)) {
 global $IN;
 $IN->QSTR = '{entry_id}';*/
?>
 {exp:comment:entries weblog="ipr_digest" sort="asc"}
 
 {comment_id}
 
 {email}
 {url}
 {ip_address}
 {comment_date format="%Y-%m-%d %h:%i:%s"}
 {gmt_comment_date format="%Y-%m-%d %h:%i:%s"}
 
 1
 
 0
 
 
 {/exp:comment:entries}

--}
 ]]>
 

 
 {exp:xml_encode}{title}{/exp:xml_encode}
 {title_permalink}-attachment
 {entry_date format='%Y-%m-%d %h:%i:%s'}
 {exp:xml_encode}{author}{/exp:xml_encode}
 
 
 
 
 2000{entry_id}
 {entry_date format='%Y-%m-%d %h:%i:%s'}
 {gmt_entry_date format='%Y-%m-%d %h:%i:%s'}
 closed
 closed
 {exp:xml_encode}{url_title}{/exp:xml_encode}-attachment
 inherit
 1000{entry_id}
 0
 attachment
 
 0
 
 

{/exp:channel:entries}

Step 4

Then simply fill in the portions of this example URL with the real parts. Parameters should be in all lowercase; I’ve capitalized them for readability.

mysite.com / my-export / CHANNEL / POST_TYPE / CONTENT_FIELD / FEATURED_IMG_FIELD

Step 5

Save the output into a file locally you get as something like “ee-import-to-wp.xml” — in a browser just use the “Save As..” option, the code above will automatically tell your browser it’s an XML file.

Step 6

Go to the WordPress Import screen located at Tools » Import area of your WordPress Dashboard (/wp-admin/admin.php?import=wordpress), select the file and click “Upload file and import”

Step 7

Follow the instructions on the screen from there, if prompted, choose to import attachments / images, this will save them into your WordPress site instead of linking to them on your ExpressionEngine site.

Enjoy

It’s time to sit back and enjoy, or you can rinse and repeat for additional post types / channels by following steps 4-7 again.

Tags

My time at WordCamp San Francisco 2012

Aug 22 2012

WordCamps have been going on for years across the globe, but where did they start? San Francisco 2006! Every year since then, San Francisco has hosted WordPress users and developers from across the world. I’ve never been to WordCamp San Francisco, or WCSF for short, but this year I decided I just couldn’t let it pass me up again. The decision to attend WCSF 2012 has turned out to have an profound effect on me..

Lesson Learned — You can only get two of these three: Features, Stability, On-time

Matt Mullenweg said this in his State of the Word 2012 keynote, and it really hit home for me. This is something that I’ve gotten wrong with my own project, the Pods Framework, of which I am the lead developer. I have spent two years developing the next major release, Pods 2.0, with features and stability as my primary focus. The problem with that? Time. It takes more time to get every single feature in, and instead of prioritizing and pushing less important features further into future releases, I focused on getting all of the features that I had wanted — all at once.

This was a mistake, disregarding any other time-related delays out of my control, this is the biggest deficiency of mine that I’ve had to learn from in my professional career. I’m used to getting projects done at work and on schedule at RD2. We have a team that handles every aspect of the project from start to finish. But when it comes to managing a project on my own — why was I so oblivious?

The problem was fear of failure. When it came to my own work, I wanted all three — Features, Stability, AND On-time — and I didn’t want to accept my own limitations. Lesson learned, and since WordCamp San Francisco 2012, Pods 2.0 has now gone beta!

My presentation: All You Can Eat Content Types

Photo by

I had the privilege and honor of being asked by Matt Mullenweg to speak at this year’s WCSF. I chose to speak about what I know best and have been studying closely for the past 6 years in my development career: Content Types. In my presentation, I went over all of the different baked in and extensible content types within WordPress.

Each of the presenters were given a tight squeeze: 15 minutes to expand the minds of the audience, and 5 minutes for Q&A while the next presenter sets up. It really was quick, I had a lot of information to cover and I went through and even skipped areas when I was pressed for time.

Please enable Javascript and Flash to view this VideoPress video.

Luckily for everyone, my presentation slides can be found on SlideShare or you can watch the video at WordPress.tv as seen above.

Dev Day: Unit Testing takes time but saves time

Photo by

WCSF had a Development Day on the following Sunday this year. WordPress developers, designers, and testers all got together to work on projects surrounding WordPress Core. There was around 80-90 people there and absolutely no shortage of talent.

Unit Testing was something I’ve always been interested in but didn’t quite understand how to take advantage of. What is Unit Testing? It’s basically a set of methods that run to check individual functions in a project. Each test checks that what a function does and returns, is as expected. If it’s not, it fails, and it’s an easy litmus test to see if a change you’ve made has greater implications. WordPress has had the great fortune of contributors putting together a Unit Testing suite for WordPress that tests against most, if not all, of it’s primary functionality.

Why does Unit Testing make sense? Throughout any project, changes are inevitable. But how do you save time testing and be able to know with a few keystrokes if that commit to a project is going to work besides testing the most common use-cases by hand? That’s where Unit Testing steps in, it’s all about saving time and not sacrificing stability for quick changes.

At RD2, there’s room for Unit Testing throughout our larger projects, especially what we’ve got in our labs currently. I’m planning on using everything I learned at WCSF Dev Day to contribute back new and updated unit tests for any patches I submit to the WordPress core and any patches I’m able to help on. Another benefit to me learning about Unit Testing is that I can now use that same knowledge to improve how Pods 2.x is developed while maintaining stability. This is an exciting area I’m really looking forward to exploring further.

It’s not WordCamp without BBQ

This should be a requirement at every WordCamp in the United States. We were all treated to a BBQ lunch with some of the best BBQ outside of Texas I’ve ever had. There was BBQ Brisket, Chicken, all the fixings, and one of my favorites – Mac and Cheese. Later on at the after party, attendees were treated to a combination of four different food trucks providing a variety of food stylings to feed any appetite throughout the night. All in all, good food promotes a really good experience overall, WCSF did this with flying colors.

Never miss Breakfast

I can’t quite remember a time that I’ve flown out of town, stayed in a hotel, and hadn’t hit the hotel breakfast buffet at least once. WordCamp San Francisco 2012 was no exception.

I am a huge fan of breakfast, and just wanted to share with you how awesome it was. I grabbed a pile of bacon, added spatulas of hash browns, grabbed a croissant, and then finished the dish with a dozen eggs. Okay maybe not a dozen, but who knows, it could have been, and it was glorious! Oh, that French toast was good too, but once I dug into the main dish, there was no comparing the two.

I shared this feast with Matt Gibbs, the founding developer of Pods who I’ve collaborated with for the past 4 years since the project got it’s start in the Fall of 2008. In fact, both of us continue to reminisce over how fulfilling the breakfast was for our code-centric souls!

Tags

, , , ,

Youtube Browser-Based Uploader with PHP

Aug 21 2012

Download Example

RD2 wanted to add something special with our 3rd iteration of building the World Traveler Internship website. We created a very unique feature within the application process. The users could upload their submission video and with a little behind the scenes magic, the video is automatically added to a designated World Traveler Internship YouTube account and added to their bio page on the website.

This is made possible using Youtube’s browser-based uploading API v2.0. The example that the API gives you are good, but not great. I wasn’t able to find examples for exactly what I needed in PHP. So I created my own 2 step process for uploading the user’s file to YouTube.

Before you start, I would suggest downloading the example and looking at that as you read the below code. Below are the steps you will need in order to create your browser-based Youtube uploader:

1. Create your 2 step form

index.php



    

        
        
        
        
        
    
 


token != '' ) : ?>
    
Title:

    

    
Description:

more…

Tags

, , , , , , ,

Honeymoon with the Sharks

Aug 17 2012

Cocos Island (), best known as the “Island of the Sharks”, is one of the most remote and beautiful islands in the world. A secluded island 350 miles off the coast of Costa Rica, it provides a  sanctuary for large underwater creatures from around the world. Why did I end up coming here for my honeymoon? Well if you want to see schools of hammerhead sharks, whale sharks, tiger sharks, manta rays, and other amazing fish, then you come to Cocos.

My journey starts off in the town of . Stepping off the airplane I could not believe how amazing the climate was here. It was 72 degrees, no humidity and a slight breeze. It was perfect weather. I later found out it was because San Jose has an elevation of 3,840 ft above sea level. The next morning, a bus picked us to take us to the dive boat. It took about 2 hours of weaving in and out of huge mountains to get to the dock. The Undersea Hunter Group has 3 vessels, of which we were aboard the the largest and newest boat called the “Argo”. It spans 130 ft, has 3 decks, and can house 14 crew and 16 divers. It carries 4 skiffs, and an undersea submersible that can take people to 1000 ft. To say the least, its a pretty unbelievable live-a-board.

Undersea Hunter Argo

So we set off into the open water. It takes around 34-36 hours to travel to Cocos Island. This voyage is not for the weary. Many of us were seasick the entire time and were detained to our bunk beds all day and night. But once we arrived that morning, the sun was shining, the sea was calm, and the beautiful island of Cocos was out our window.

more…

Tags

, ,

« Previous Posts