How to import your IIS logs into Piwik

Piwik is pretty awesome if you’re in the market for the functionality it provides. Unfortunately a lot of people who use IIS, SharePoint, .NET are not familiar with Python and may struggle with getting their IIS logs into Piwik. This should help if you are in that situation.

  1. Move your logs to a central location. PowerShell as a scheduled job is perfect for this. In this example let us assume it is D:LOGS
  2. Let us also assume that you are running Piwik on the local machine aka localhost on port 1000 and the ID of the site in Piwik is 1. These values are in the code below.
  3. Create a script called importFiles.py and paste the below into it. The TAB indents are pretty important.

import os, fnmatch, subprocess

def find_files(directory, pattern):
    for root, dirs, files in os.walk(directory):
        for basename in files:
            if fnmatch.fnmatch(basename, pattern):
                filename = os.path.join(root, basename)
                yield filename

for filename in find_files(‘C:LOGS’, ‘*.log’):
    #print (filename)
    os.system(“C:/inetpub/wwwroot/Piwik/misc/log-analytics/import_logs.py –url=http://localhost:1000/ ” +str(filename) +” –idsite=1″)

Notes about the above.

  • You must NOT install Python 3.x. Install 2.x. This is a compatibility issue with Piwik.
  • I use MariaDB. If you are not familiar with it think of it as simply as it being a fully compatible fork of MySQL.
  • You can definitely optimize this script further by reading the Piwik docs. This is about as basic as it can be and still get the job done.
  • This is recursive so if you are moving the logs into server named subfolders, a good idea if you have a farm, it should work just fine.
  • I would strongly recommend that you move the files to a “processed” folder outside of the path above once you’re done. 
  • Make sure to set index.php as your default document in the IIS site you are running Piwik from.
If you need to do over then you need to get the log file data out of the DB. to do so remove the appropriate tables and delete the appropriate values similar to below. More details are in the Piwik FAQ here.
DROP TABLE piwik_archive_numeric_2011_01, piwik_archive_numeric_XX;
DELETE FROM piwik_log_visit WHERE idsite = X;
DELETE FROM piwik_log_link_visit_action WHERE idsite
= X;
DELETE FROM piwik_log_conversion WHERE idsite
= X;
DELETE FROM piwik_log_conversion_item WHERE idsite
= X;
More info and relevant downloads can be gotten here: 

Browser detection: solving the quandary with a bit of PHP

The browser and its specific version can tell you a lot about your visitors. Not just that they are running OpenSolaris, Windows 7, or Chrome OS but much, much more. It also can help you enable, or more likely disable, certain features of your site, content, or application. Therefore it is more than critical to determine it correctly. It is/should be mandatory.

So what are your options. Precisely three.

  1. Do nothing
  2. Go client side
  3. Go server side

1 is not worth mentioning, 2 is typically performed by JavaScript which leaves 3.

Now leaving aside the potential impact on your caching mechanism/plans/shenanigans option 3 can bring some great benefits without sending the client browser any payload. But if you have to do it yourself it can be a royal pain as browser versions and new products are like beer. Everybody will try a new one but usually not for long. (that may need work.) So what to do? Cue Browser.php


Browser.php is a PHP class that helps this process by collecting data about a visitor’s browser such as:

  • platform used
  • browser
  • browser version

It is very easy to implement.

Typical usage

$browser = new Browser();
if( $browser->getBrowser() == Browser::BROWSER_FIREFOX && $browser->getVersion() >= 2 ) {
echo 'You have FireFox version 2 or greater';
}

Needless to say you must do the usual beforehand such as:
require_once(‘Browser.php’);
and
$browser = new Browser();

Resources: http://chrisschuld.com/projects/browser-php-detecting-a-users-browser-from-php/

PHP: Finding the binary

The other day I was stumped, albeit rather briefly, by something I should not have been. I was scripting out some CRON jobs for a CRM system when I found out I could not locate where the hosting company had put the PHP binary. Now that is something you usually do not have to think about so it can easily fall into the ‘been a while’ category.

So if you ever find yourself asking, where is the PHP binary? or just where is PHP? Here are some steps to follow.

Telnet/Shell
If you have shell access via telnet, log in, and type “which php” and you should be given the location to use.

If you don’t have shell, try one of the following:

Create a php page with just the following: and then call the page, this should tell you the location of PHP.

echo system('which php');

If that does not work, though it should, try checking one of the paths listed under path in the ENVIRONMENT section of the output with this.Take note though that this will give you the environment path so you will still need to add /php to it…

echo phpinfo();

How to install MemCached with Apache, PHP, and MySQL on FreeBSD

memcached is a high-performance, distributed memory object caching system, generic in nature, but intended for use in speeding up dynamic web applications by alleviating database load.

Danga Interactive developed memcached to enhance the speed of LiveJournal.com, a site which was already doing 20 million+ dynamic page views per day for 1 million users with a bunch of webservers and a bunch of database servers. memcached dropped the database load to almost nothing, yielding faster page load times for users, better resource utilization, and faster access to the databases on a memcache miss.

Notes

  • There are definitely steps in this that can be changed, extended, other, but these will get you what you want in a good learning fashion. And learning is key!
  • You can just as easily install memcached on an IIS server.
  • Yes you can download binaries with pkg_add etc. but building will help you literally see what is going where. One just has to make sure that some housekeeping is performed during the build process. This is done with the clean parameter in the make command.
  • This appears to be quite a popular post. Please keep in mind that version numbers change so update the strings below.
To Start

First off build a FreeBSD box. It’s easy, fast, and best of all free once you have the hardware – real or virtualized. Don’t forget to install the ports. You’re going to need them going forward.

For dependency issues we’re going to install all the items in this order:

  1. MySQL Server 5.x
  2. Apache 2.x
  3. PHP5
  4. PDF-Lite
  5. PHP Extensions
  6. MemCached

1: MySQL

Move to the ports folder, specify some build parameters that will helps speed things up a tad.
cd /usr/ports/databases/mysql51-server
make BUILD_OPTIMIZED=yes BUILD_STATIC=yes
make install clean
Edit /etc/rc.conf and add mysql_enable=”YES”
Apply the configuration file: cp /usr/local/share/mysql/my-medium.cnf /etc/my.cnf
Assign the master password: /usr/local/bin/mysqladmin -u root password ‘new-password’
Start MySQL: /usr/local/etc/rc.d/mysql-server start

2: Apache

cd /usr/ports/www/apache22
make install clean
Edit /etc/rc.conf and add apache22_enable=”YES”
Edit /boot/loader.conf and add accf_http_load=YES

3: PHP

cd /usr/ports/lang/php5
make install clean  !!In the popup at the least check off “Build Apache Module”
4: PDF Lite
fetch ftp://ftp.swin.edu.au/gentoo/distfiles/PDFlib-Lite-7.0.2.tar.gz
sudo mv PDFlib-Lite-7.0.2.tar.gz /usr/ports/distfiles/

5: PHP Extensions
cd /usr/ports/lang/php5-extension 
make install clean !!Confirm that sockets is selected.
Apply the configuration file: cp /usr/local/etc/php.ini-dist /usr/local/etc/php.ini
Enabling Apache to run PHP
Edit the Apache configuration file /usr/local/etc/apache22/httpd.conf and add
AddType application/x-httpd-php .php
AddType application/x-httpd-php-source .phps
Change DirectoryIndex index.html to DirectoryIndex index.php index.html
Start Apache /usr/local/etc/rc.d/apache22 start
6: MemCached
cd /usr/ports/databases/pecl-memcache
make install clean
cd /usr/ports/databases/memcached
make install clean
Edit /etc/rc.conf and add memcached_enable=”YES”
Start MemCached with /usr/local/bin/memcached -d -u nobody
Confirming MemCached is running
Run netstat -an to confirm that a process is listening on TCP 11211
It may also be useful to run /usr/local/bin/php -i | grep -i ‘memcache’

MongoDB: installing on windows as a service and your first database

MongoDB is an document-oriented DBMS. Think of it as MySQL but JSON (actually, BSON ) as the data model, not relational. There are no joins. If you have used object-relational mapping layers before in your programs, you will find the Mongo interface similar to use, but faster, more powerful, and less work to set up. In fact it is so easy to get running that I’m going to show you how to get it up and running as a Windows service in a few minutes.

But first, why would you ever want to use MongoDB?

The goal of MongoDB is to bridge the gap between key-value stores (which are fast and highly scalable) and traditional RDBMS systems (which provide rich queries and deep functionality). MongoDB is designed for problems that aren’t easily solved by traditional RDBMSs, including problems that require databases to span many servers.

MongoDB is a document-oriented database. This means that unlike a relational database management system, MongoDB manages collections of JSON-like documents. This allows many applications to model data in a more natural way, as data can be nested in complex hierarchies and still be query-able and indexable.

MongoDB supports an automated sharding architecture, enabling horizontal scaling across multiple nodes. For applications that outgrow the resources of a single database server, MongoDB can convert to a sharded cluster, automatically managing failover and balancing of nodes, with few or no changes to the original application code.

Moving on…

1: Getting the binaries…
Download the windows version here and be sure to get the 64bit version. The 32bit version has a limit of about 2.5Gb whereas the 64bit version has almost none… The reason for this is that the MongoDB storage engine uses memory-mapped files for performance and the goal for the engines code was to keep it as uncluttered as possible.

2: Create some folders
Before opening the downloaded archive you have to create a folder with content at the root of C: called “data”. Inside that create a folder called “db” Note that from now on I’m going to be giving the equivalent of shell commands highlighted and as you continue to read you will see why.

Start -> Run -> CMD
C:
cd C:
mkdir data
mkdir datadb
mkdir datalogs

At this point you should now have this structure on your file system.

3: Extract the downloaded files
You should have downloaded something named like this “mongodb-win32-x86_64-1.4.4.zip”

Extract the archive and move its contents to C: so you should see this type of structure:

4: Set MongoDB as a Windows Service
Looking inside of “bin” we can see a few .exe files. Our focus for now is going to be on “mongod.exe”

Setting it up as a service requires a couple of decisions. At a minimum:

  1. Where shall the logs go? In this example c:mongologs
  2. Where shall the data go? In this example c:mongodata 
  3. What port do want to run the web service on? In this example 1001.

With that decided creating the service is as easy as running the below from the command line.

mongod –bind_ip 127.0.0.1 –logpath c:datalogs –logappend –dbpath c:data –directoryperdb -port 1 –install

! If you wish to remove it just use: mongod –remove

So now Mongo should be visible in Services

Note the path to the executable and the fact that the service is not yet started…

Start the service and go here: http://127.0.0.1:1001/ where you should see the following information.

5: Accessing database
Accessing the database is done via mongo.exe found in bin
Run
C:
cd mongodbbin
mongo

and you should now be in the administrative shell like this.

! If you get an error like below just remove the service and redo without the port definition.

MongoDB shell version: 1.4.4
url: test
connecting to: test
Tue Jul 13 14:44:59 JS Error: Error: couldn’t connect: couldn’t connect to server 127.0.0.1 127.0.0.1:27017 (anon):952
Tue Jul 13 14:44:59 User Exception 12513:connect failed

Saving and retrieving is as easy as below.
db.foo.insert( { a : 1 } )
db.foo.find()

I have also included what the file systems should look like as well for this example.



And that’s it. You now have a working MongoDB service on your Windows machine!
Things to note

  • A performance environment would not have everything on C: 
  • Run binmongd -h for more options, there are not that many…
  • Defining the port rather than running it on the default 27017 requires some extras. Expect errors if you do not read up a bit more on MongoDB.
  • Did you notice that the administrative shell is a JavaScript interface..?

Resources

Another easy way to perform instant and/or automated backups of MySQL

PhpMyAdmin is great for what it does but it does not excel everywhere. If you’re looking to create instant or automated backups of MySQL databases consider Sypex Dumper 2.x

Built with PHP, it has an Ajaxed interface and can run database restores as well. Handily it can avoid PHP script timeouts by pausing the jobs. And being licensed under the BSD license, it is absolutely free. Interestingly supposedly it can also be integrated into third-party products… Something I have not done or tried. What impressed me mostly about it was how easy interacting with databases via the browser became with it.

http://sypex.net/en/

Speed up your PHP like Facebook does with HipHop for PHP

It’s a fact that script based languages and frameworks slow you down. PHP and its huge amount of frameworks is no exception to that rule. Until you realize that it is used on the busiest site out there – Facebook.
Scripted languages typically are embraced as they have advantages in being simple to learn, code, and debug while counterbalanced with a definite sacrificing of speed. On the other side of the spectrum, compiled languages usually have a steeper learning curve and are much more strict; counter balanced with the clear fact that they are considerably faster and better optimized for handling memory. Facebook seems to have found a balance between both worlds with HipHop for PHP.
Facebook developers write lots of their core services in other languages, and then PHP is used as the glue that queries those services and builds the actual pages itself. Still, they have found that that really isn’t fast enough for what they want, so they went and wrote their own compiler, or rather an optimizer, called HipHop, which compiles PHP down to C++ which can then be compiled by G++.
You can download HipHop from here: http://github.com/facebook/hiphop-php

If you’re interested in using/trying some PHP frameworks you can start by looking at:

After The Deadline is a project of Automattic (the creator of WordPress) which checks:

  • spelling
  • grammar
  • styles (finds complex phrases, cliches, bias-language & suggests simple ones)
  • and detects misused words (has a database of 1500 most misused words & suggestions)

The software used to come as a plugin for several projects (like WordPress, RoundCube, PHP List) but it is now open source and anybody can embed it to their own code.

Besides the artificial intelligence it already has, it is possible to add new rules & improve the application.

The project also has libraries for jQuery, Ruby & Python to make developing new plugins easier.


Demo: http://www.polishmywriting.com/

Mockup tool: Balsamiq Mockups

A very, very useful tool and worth the cost, $79 USD is Balsamiq Mockups. It works because Balsamiq Mockups intentionally uses hand-drawn UI elements, so that people don’t get attached to “that pretty color gradient” or think that your mockup has actual code behind it and is “practically done”…