ejes consulting

Techincal Consulting Design and Automation

Archive for August 2009

Google Scripts without Google API

leave a comment »

A client I just finished working with had a few applications where they would judge the deployment of a product (a piece of software for example) by using googles output.

They considered any search product with <100,000 results obscure, less than 1,000,000 moderate, 1,000,000 to 100,000,000 significant and greater than that still as broad.

They were doing this by hand. 

Searching google and looking at the “Results x- x of about xxxx” section of google search, and using this output to judge the deployment, as I said.

The problem here is that they wanted to script this, but NOT by using google’s api (for legal reasons).

I therefore would have to find a way around. 

The easiest way, I could think of, is just capturing the output of the search… But how? 

The command line browser “lynx” in unix systems is amazing for this sort of thing.  Lynx can render a simple html page and dump it as text (using the -dump parameter) 

All I had to do then is wrap it in a nice script that would read the rendered output, and output something meaningful.

You can find the script in my script section called “googleit.py” and “deployment.py”

Advertisements

Written by ejes

August 24, 2009 at 12:24 pm

Restrict a 14 year old from the internet

leave a comment »

So I’ve taken a bit of time off – to spend with my son who came to stay with me from the far away place called Orillia. 

I’m quite liberal with our computing environment – so he gets his own computer and to be “self regulated” over the Internet.  He knows well enough that I have quite a complex network and know all the traffic that travels through our up link to the Internet.

Giving your 14 year old unrestricted internet access has it’s ups and downs.  One major down was that he managed to eat my entire Rogers High Speed Internet quota in a matter of 4 days.

That’s right 4 days.

So to limit him, I entered his MAC address in my deny list and he isn’t able to attach to the internet.  Easy right?  No.

I also have a Wii, which has an Internet connection, and frequently will download updates for itself.  My son also loves to play the Wii.

I don’t know exactly where he got the idea, but Suddenly I noticed that my Wii was unplugged, and instead the “Wii” was browsing the Internet, Gigabytes at a time on World of Warcraft, Runescape, uTorrent, and Youtube.   A browsing pattern I knew FAR too well.

He went into the Wii’s menu, looked up it’s MAC address, unplugged  the Wii, then changed his own MAC address to the Wii’s easily circumventing my restrictions.

Like father like son, I suppose.  Now I have to physically log into my OpenBSD server from wherever I am and forcibly unload it’s networking modules with a cron job to reload them when he would be allowed to regain access to the internet.

Written by ejes

August 18, 2009 at 1:49 pm

Posted in Commentary

Tagged with

Tutorial: cvs server on FreeNAS

with one comment

First, when I started to research if “cvs” would work on my FreeNAS I found a whole lot of links to cvsd… but I recall using cvs over ssh for many many projects. Couldn’t I run just cvs over ssh?

Yes You Can!!! And this makes cvs on FreeNAS easier than ever. Not much modification is required, and I am very happy to report that I’ve got mine working very well.

So let’s get into it shall we?

In order to get cvs working, we need the cvs binary. FreeNAS is a FreeBSD core underneath the covers. There are many many releases of FreeBSD so we should find out what release we have. We can do this by running uname on the command prompt of your FreeNAS.

Mine says:

nas:/mnt/default# uname -a

FreeBSD nas.ejes.gotdns.org 6.4-RELEASE-p3 FreeBSD 6.4-RELEASE-p3 #0: Sat Apr 18 22:17:59 UTC 2009     root@vmbsd64i386:/usr/obj/freenas/usr/src/sys/FREENAS-i386  i386
nas:/mnt/default#

Now that I have the release number, I can simply browse the FreeBSD repository at ftp://ftp.freebsd.org (the 6.4 release is ftp://ftp.freebsd.org/pub/FreeBSD/releases/i386/6.4-RELEASE/6.4-RELEASE/packages/All/), and following the directory tree you can find most any other release.

Simply download the “cvs” binary cvs+ipv6-1.11.17_1.tbz (ftp://ftp.freebsd.org/pub/FreeBSD/releases/i386/6.4-RELEASE/6.4-RELEASE/packages/All/cvs+ipv6-1.11.17_1.tbz)

I used fetch on my system:

nas:/mnt/default# fetch ftp://ftp.freebsd.org/pub/FreeBSD/releases/i386/6.4-RELEASE/6.4-RELEASE/packages/All/cvs+ipv6-1.11.17_1.tbz cvs+ipv6-1.11.17_1.tbz
                        100% of  442 kB  266 kBps

Then, in the GUI, I added a group called “cvs”, I gave it the group id 2401 because port 2401 was the cvs server port (and I may want to use pserver one day).

The Group I used for my cvs

The Group I used for my cvs

I added a new user _cvs with the userid of 2401 (for the same reason my group is 2401), his home directory will be my cvs root directory. add user _cvs (id 2401), his home directory is where my cvs root will be as well i put mine in /mnt/default/cvs.

the user _cvs for myself

the user _cvs for myself

Thankfully a tbz file means a tar bzipped file. So we can use tar to extract the package from freeBSD i want all my cvs stuff in one place so i’m going to extract it to my /mnt/default/cvs/tmp directory.

nas:/mnt/default/cvs/tmp# tar -vxzf cvs+ipv6-1.11.17_1.tbz
x +CONTENTS
x +COMMENT
x +DESC
x +MTREE_DIRS
x man/man1/cvs.1.gz
x man/man5/cvs.5.gz
x man/man8/cvsbug.8.gz
x bin/cvs
x bin/cvsbug
x bin/rcs2log
x share/cvs/contrib/README
x share/cvs/contrib/check_cvs
x share/cvs/contrib/clmerge
x share/cvs/contrib/cln_hist
x share/cvs/contrib/commit_prep
x share/cvs/contrib/cvs2vendor
x share/cvs/contrib/cvs_acls
x share/cvs/contrib/cvscheck
x share/cvs/contrib/cvscheck.man
x share/cvs/contrib/debug_check_log
x share/cvs/contrib/intro.doc
x share/cvs/contrib/log
x share/cvs/contrib/log_accum
x share/cvs/contrib/mfpipe
x share/cvs/contrib/pvcs2rcs
x share/cvs/contrib/rcs-to-cvs
x share/cvs/contrib/rcs2log
x share/cvs/contrib/rcslock
x share/cvs/contrib/sccs2rcs
x info/cvs.info
x info/cvs.info-1
x info/cvs.info-10
x info/cvs.info-2
x info/cvs.info-3
x info/cvs.info-4
x info/cvs.info-5
x info/cvs.info-6
x info/cvs.info-7
x info/cvs.info-8
x info/cvs.info-9
x info/cvsclient.info
x info/cvsclient.info-1
x info/cvsclient.info-2
x info/cvsclient.info-3
nas:/mnt/default/cvs/tmp#
The only binary we really need in that entire mess is the actual “cvs” binary. Let’s copy it to /mnt/default/cvs/bin.
Once I’ve done that, I can symbolically link the cvs binary to a better place in your path, i put it in /bin. /bin/ln -sf /mnt/default/cvs/bin/cvs /bin/cvs
Since I want this to happen each time after boot, I put it in the System|Advanced|Command scripts section of the FreeNAS GUI.
symlink cvs to /bin
While I’m there I should fix the permissions of /tmp so that everyone can write in it. chmod a+rwx /tmp works nicely.
repermission /tmp
Don’t forget to run them post-init (and run it too).

Go back and “Give full shell access to user.” to the _cvs user and log-in or “su _cvs” to become him.

Now we have to initialize the cvs root directory; mine is /mnt/default/cvs/root. Run cvs init, and we’re off.

nas:/mnt/default/cvs/tmp# su _cvs
%cvs -d ~/root init
%exit
nas:/mnt/default# chown -R _cvs:cvs cvs
nas:/mnt/default#

Everything should be functional, all we need to do now is add anyone we want to be able to access cvs, to the “cvs” group. I added “user” as a test.

On my OpenBSD terminal I setup a simple cvs system

$ CVS_RSH=ssh cvs -d:ext:_cvs@nas:/mnt/default/cvs/root checkout .

works! GREAT!! Add my user to the “cvs” group, remove “shell” access from _cvs done!

My Remote users have to set these settings:

CVS_RSH=ssh

CVSROOT=:ext:user@host:/mnt/default/cvs/root

CVSEDITOR=nano

Now I can import directories using: cvs import -m “comment” module_name committer start

Check them out on another machine using: cvs checkout module_name

Add files or directories: cvs add “file/directory”

And Commit my changes once they’re made: cvs commit

For example:

nas:/mnt/default# ssh user@192.168.0.2
user@192.168.0.2's password:
Last login: Tue Aug  4 16:46:50 2009 from 192.168.0.3
OpenBSD 4.4 (GENERIC) #1021: Tue Aug 12 17:16:55 MDT 2008
Welcome to OpenBSD: The proactively secure Unix-like operating system.
Please use the sendbug(1) utility to report bugs in the system.
Before reporting a bug, please try to reproduce it with the latest
version of the code.  With bug reports, please try to ensure that
enough information to reproduce the problem is enclosed, and if a
known fix for it exists, include that as well.
$ CVS_RSH=ssh; export CVS_RSH
$ CVSROOT=:ext:user@192.168.0.3:/mnt/default/cvs/root; export CVSROOT
$ CVSEDITOR=nano;export CVSEDITOR=nano
$ mkdir src
$ cd src
$ mkdir testing
$ cd testing
$ touch a file
$ ls
a    file
$ cvs import -m "testing" test ejes start
user@192.168.0.3's password:
N test/a
N test/file
No conflicts created by this import
$ mkdir new
$ cvs add n
$ touch new/more
$ cd ..
$ ls
testing
$ rm -rf testing/
$ cvs checkout test
user@192.168.0.3's password:
cvs checkout: Updating test
U test/a
U test/file
$ cd test
$ mkdir new
$ touch new/more
$ cvs add new
user@192.168.0.3's password:
? new/more
Directory /mnt/default/cvs/root/test/new added to the repository
$ cvs add new/more
user@192.168.0.3's password:
cvs add: scheduling file `new/more' for addition
cvs add: use 'cvs commit' to add this file permanently
$ cvs commit
cvs commit: Examining .
cvs commit: Examining new
user@192.168.0.3's password:
RCS file: /mnt/default/cvs/root/test/new/more,v
done
Checking in new/more;
/mnt/default/cvs/root/test/new/more,v  <--  more
initial revision: 1.1
done
$

Written by ejes

August 4, 2009 at 5:44 pm