How to compare two Excel / CSV / XLSX files using PowerShell

The following PowerShell script allows you to compare ALL the columns in two CSV files contents against each other.

cls

# Import the files
 $file1 = import-csv -Path "file1.csv";
 $file2 = import-csv -Path "file2.csv";

# Get the list of properties
 $props1 = $file1 | gm -MemberType NoteProperty | select -expand Name | sort | % {"$_"}
 $props2 = $file2 | gm -MemberType NoteProperty | select -expand Name | sort | % {"$_"}

if(Compare-Object $props1 $props2) {

    # Check that properties match

    throw "Properties are not the same! [$props1] [$props2]"

} else {

    # Pass properties list to Compare-Object

    "Checking $props1"

    Compare-Object $file1 $file2 -Property $props1

}

Creating symbolic links in FreeBSD

Sometimes when you FTP into a FreeBSD environment you need to bounce around to directories acroos the file system. One way to simply this process when working in a FTP client is to create a symbolic link as a file. It is really easy.

To go from point A to point B where point A is a click able file and B is a target folder just run:

ln -s /path/to/folder/B /where/you/want/the/file/to/exist/B

Linux 2.6.34 has been released…

Linux 2.6.34 has been released. This version adds two new filesystem, the distributed filesystem Ceph and LogFS, a filesystem for flash devices. Other features are a driver for almost-native KVM network performance, the VMware balloon driver, the ‘kprobes jump’ optimization for dynamic probes, new perf features (the ‘perf lock’ tool, cross-platform analysis support), several Btrfs improvements, RCU lockdep, Generalized TTL Security Mechanism (RFC 5082) and private VLAN proxy arp (RFC 3069) support, asynchronous suspend/resume, several new drivers and many other small improvements. See the full changelog here.”

How to: Recover a raw disk image of your OS X Filevault. and then read it’s contents…

File Vault is the encryption mechanism used to protect user accounts on an OS X file system. Disabled by default it has become more popular since originally released with 10.3 “Panther.”

When enabled, FileVault mounts and unmounts encrypted file systems duringthe user logging in and out process. In 10.5 “Leopard” this image is a Sparse Bundle, recognizable as a collection of 8Mb files. A change from before where a single and large file was used. One primary reason for the breakup to 8Mb is to allow Time Macine backups. A note to upgraders though, be sure to disable and then re enable Filevault otherwise you’ll end up with that sparse image format.

The user’s home directory is encrypted using AES with a key derived from the user’s login password. Content is automatically encrypted and decrypted on the fly. However, this data is not bound as tightly as some assume. Nor is the housekeeping as good as others would assume. Succinctly put: “Encrypted volumes preserve much of the data that is deleted from the volume. Because a file vault is treated as a separate file system, a free space wipe does virtually nothing to destroy deleted data stored inside a vault. Due to Apple’s policy of “security by obscurity”, many false assumptions have been made about the File Vault encryption mechanism, potentially exposing sensitive data to someone with the right tools. Users should be aware of File Vault’s caveats and limitations before relying on it as a means of securing data.

Here are the surprisingly easy steps to pull together a raw disk image from a users FileVault. This particular users name being “beer”. There is an assumption that you already have the key… If you’re looking to crack the encryption look elsewhere for vfcrack, cold boot attacks, and others. This image is suited for FTK or EnCase processing or carving up with Scalpel or Foremost…

Step 1
$ sudo -s
# ls -l /Volumes/Suspect_Disk/Users
total 0
-rw-r–r– 1 root wheel 0 Sep 23 2007 .localized
drwxrwxrwt 4 root wheel 136 Oct 5 18:34 Shared
dr-x—— 3 beer staff 102 Oct 10 13:11 beer

Step 2
# ls -l beer
total 0
drwx——@ 6 beer staff 204 Oct 10 12:58 beer.sparsebundle

Step 3
# hdid -readonly beer.sparsebundle

After being prompted for the File Vault password, the hdid tool will give you a mapping to a series of raw devices for the image:

/dev/disk2 Apple_partition_scheme
/dev/disk2s1 Apple_partition_map
/dev/disk2s2 Apple_HFS

Step 4
The third device down, disk2s2, is the one you’re interested in; it contains the HFS file system containing the home directory. You can copy off the enire file vault’s contents with a simple disk copy:

# dd if=/dev/rdisk2s2 of=/Volumes/Digital_Vault/rdisk2s2 bs=16384

Here’s the annoying part: because the vault is mapped onto the file system, you’re actually going to get a very large file containing a lot of padding; this could be several hundred gig in size depending on the virtual size of the file vault. You’ll need to make sure you have adequate disk space to contain this image.

Once you’ve got the image, it can be accessed much like any other HFS image. You can load it into FTK, or use a data carving tool like Scalpel to get data off of it. In addition to this, you can rename the image to have a .dmg extension and mount the decrypted file system right on your mac:

# mv /Volumes/Digital_Vault/rdisk2s2 /Volumes/Digital_Vault/rdisk2s2.dmg
# hdid -readonly /Volumes/Digital_Vault/rdisk2s2.dmg /dev/disk3/Volumes/beer

And that’s pretty much it…

Update on ntfs-3g

Back in December I posted about setting up direct read and write access to a NTFS drive from 10.5. It all seemed to be working okay until last week when I had to move a couple of VHD files, close to 500Gb, from a Mac running 10.5 across the wire to a windows based NAS. Good grief is all I can say about sustained performance. It took days to complete. More like a week to be honest… Why, as of yet I do not know but there is defintely something “up” with either the Mac or the driver. As the NAS is “fine.”

dd: clean your drive securely

Now like anybody I’m a BIG fan of wiping old drives using dd but sometimes there’s a tool out there that will do most if not all of the work for you. Cue DBAN. OR as the site says:

Darik’s Boot and Nuke (“DBAN”) is a self-contained boot disk that securely wipes the hard disks of most computers. DBAN will automatically and completely delete the contents of any hard disk that it can detect, which makes it an appropriate utility for bulk or emergency data destruction.

Complemented with TrueCrypt you will have a mighty secure setup. Possible / definite paranoia issues too… But your data will be secure. For the more command line orientated the old reliable dd if=/dev/urandom of=/dev/disk bs=1k is good enough imho. (It puts random bits in place as opposed to a regular pattern. Not that it will stand up to NSA level scrutiny but it’s more than enough for most data recovery…)

For more go to:

DBAN: http://www.dban.org/
TrueCrypt: http://www.truecrypt.org/

When did you last check your backups?

I am oft called “paranoid”, which I prefer to think of more as “highly aware”, when it comes to backups. The more baskets you have, the better your chances when you need to catch something. STSADM export function is your friend. Use it, test it, and do so frequently (I do it with virtual environments.) Because when the fan starts going chunka-chunka you do not want to be left wondering anything more than how many hours downtime are coming. Case in point, about not testing your backups not SharePoint / STSADM, Journal space literally evaporated this week…

Journalspace is no more.
DriveSavers called today to inform me that the data was unrecoverable.
Here is what happened: the server which held the journalspace data had two large drives in a RAID configuration. As data is written (such as saving an item to the database), it’s automatically copied to both drives, as a backup mechanism.
The value of such a setup is that if one drive fails, the server keeps running, using the remaining drive. Since the remaining drive has a copy of the data on the other drive, the data is intact. The administrator simply replaces the drive that’s gone bad, and the server is back to operating with two redundant drives.
But that’s not what happened here. There was no hardware failure. Both drives are operating fine; DriveSavers had no problem in making images of the drives. The data was simply gone. Overwritten.
The data server had only one purpose: maintaining the journalspace database. There were no other web sites or processes running on the server, and it would be impossible for a software bug in journalspace to overwrite the drives, sector by sector.
The list of potential causes for this disaster is a short one. It includes a catastrophic failure by the operating system (OS X Server, in case you’re interested), or a deliberate effort. A disgruntled member of the Lagomorphics team sabotaged some key servers several months ago after he was caught stealing from the company; as awful as the thought is, we can’t rule out the possibility of additional sabotage.
But, clearly, we failed to take the steps to prevent this from happening. And for that we are very sorry.
So, after nearly six years, journalspace is no more.
If you haven’t yet, visit
Dorrie’s Fun Forum; it’s operated by a long-time journalspace member. If you’re continuing your blog elsewhere, you can post the URL there so people can keep up with you.
We’re considering releasing the journalspace source code to the open source community. We may also sell the journalspace domain and trademarks. Follow us on twitter at
twitter.com/jsupgrades for news.”

OpenSolaris 2008.11: Time Slider



Still suffering from lackluster reception outside of the Sun/Solaris community, OpenSolaris 2008.11 was released today. Leaving aside the fixes in since 2008.05 there is one exceptional feature worth spending a moment on. Time Slider. 

In my opinion ZFS is one of the most impressive computer technologies out there. Period. Amongst other things it has very advanced storage pooling and support for deep snapshotting. As with anything else, with a core technology you can only succeed when you expose it effectively to end users and this version of OpenSolaris shows that the developers are certainly heading in the right direction.

Time Slider is literally enabled with a click in the Administration menu. Once enabled Nautilus has a slider representative of points in time and it just works.
More details can be found here: http://opensolaris.org/os/

Take note that there’s a live cd as well!!!

Other fixes/updates…

  • ZFS Time Slider and Songbird 
  • Suspend/Resume and CPU power management 
  • Distribution Constructor and Prototype Automated Installer 
  • WebStack with 64-bit MySQL, CherryPy, and DTrace for Ruby 
  • GNOME 2.24, OpenOffice 3.0, and Firefox 3