pSearch Source!!

So I had some trouble getting my source put onto wordpress.  I can understand their point, they don’t want to share .zip, .tar or any other archive container formats.

In the intrest of brevity, I decided to just use a free file host.  I chose medifire, it was top on google when I checked.

http://www.mediafire.com/?u01bf1nwbemata9

There is where you can find the historical archives of my search development.  I did my best to ensure that it could be compiled on Windows (32-bit XP via, MinGW) or Linux (ubuntu 64-bit server), sometimes OpenBSD. You’ll likely need sqlite (http://www.sqlite.org/) and libcurl (http://curl.haxx.se/), you’ll probably need pcre libraries as well (http://www.pcre.org/).  If you try to compile something that looks like it should work, let me know and I’ll see if there’s any libraries that I might be missing, or at least I can let you know if it SHOULD compile.

All the above source is released under the original BSD Licence.

Copyright (c) 2010-, Evan Stawnyczy (ejes consulting) ejes@torfree.net
All rights reserved.

Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are met:
1. Redistributions of source code must retain the above copyright
   notice, this list of conditions and the following disclaimer.
2. Redistributions in binary form must reproduce the above copyright
   notice, this list of conditions and the following disclaimer in the
   documentation and/or other materials provided with the distribution.
3. All advertising materials mentioning features or use of this software
   must display the following acknowledgement:
   This product includes software developed by ejes consulting.
4. Neither the names ejes consulting, Evan Stawnyczy nor the
   names of its addional contributors may be used to endorse or promote
   products derived from this software without specific prior written
   permission.

THIS SOFTWARE IS PROVIDED BY EVAN STAWNYCZY AND EJES CONSULTING ''AS IS''
AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
ARE DISCLAIMED. IN NO EVENT SHALL <COPYRIGHT HOLDER> BE LIABLE FOR ANY
DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

As of right now, I’m working on a complete rewrite.

This rewrite I’m hoping will serve as good working reference code.  It will be clear, easy to understand and most likely SLOW.  I am now using ODBC instead of limiting users to sqlite.  This also will hopefully allow enterprises to adopt use without too much trouble.  I decided to scrap the web-server aspect – hopefully someone will want to bundle search and web-server for home use.  Especially with the wide adoption of ipv6 a single workstation could easily share information with others all indexed through a private search network.  At the very least someone should write an nginx (http://nginx.org/), lighttpd (http://www.lighttpd.net/) and apache (http://www.apache.org/) module that indexes static and cached content and publishes the search results.

That leads me to trying to build this as an ip-agnostic application. I want it to run in both an ipv4 and ipv6 network. Of course this has it’s own challenges as well.  I’m trying to maintain ANSI compliance where I possibly can so that it can be easily portable, and mostly so that it can run on windows or unix without too much trouble.

pSearch – a peer to peer, distributed search engine

Forward

So I haven’t been posting very much for the last while, and this is mainly because I’ve been very busy.

I always have several projects on the go, and I don’t have enough time to devote to all of these things at once, so usually the least interesting project gets placed on the back burner.

That is what happened to this blog.

Now I’ve spent a great deal of time on this, and have produced some very good design documents as well as a bunch of source code.  So… Without further ado

This is my distributed, peer-to-peer search engine.

Attached to this post you’ll find a couple of architecture documents, a pdf with a visual diagram of how this engine is suppose to work, and another pdf with a long winded, half written description of why and how I expect this conceptually to run.

I’m not a writer, and am mostly a technical person, however, I am actively updating and modifying this project so expect updates as it goes.

The first document is the “pSearch – Document“

In this document I attempt to explain the strategy, and reasons for this project and what  I hope that it will accomplish.   This document is incomplete, but I encourage you to read it anyway.

The second document is the “pSearch – Drawing“

In this document I have detailed the major aspects of the distributed search.  Hopefully it’s easy to follow, I don’t expect this diagram to change very much.

And I have a LOT of source code that I still have to organize – much of it will be posted here and some of it is too embarrassing.

Summary

So, without drudging into my documentation in too much detail (I posted them above, feel free) a simplified “how does this work” seems appropriate.

Each peer will accept connections  from the internet.  Each search request is forwarded to other peers as defined in it’s database.

While this happens, it also uses a second task to search it’s own internal database.  On a private home machine this internal crawler has a small collection of sites and keywords based on several configurable data collection points (such as your browser cache, or installed programs) which would automatically include a lot of data that would be specific to you.   A public internet site would index their own pages (this isn’t mandatory, but preferred).

After that, it’s a simple case of matching the keyword and publishing the results to the connected client.

Peers who respond quickly, and with a lot of results are flagged as “experts” when it comes to this set of terms.  This way, when you search for a similar set of terms again, the “expert” peers will be consulted first.

This way, common search terms will be responded to by clients who have a lot of information on these terms.  For example a site that indexes movies (like imdb) would respond with a lot of results for movie titles and information about films, but probably have very little to respond when a query has some specific request about cars.

Expect more as I develop more.  I encourage anyone to read and comment about my designs.

Javascript Obfuscation to the MAX!!!

I was stumbling around the internet today and found this:

($=[$=[]][(__=!$+$)[_=-~-~-~$]+({}+$)[_/_]+
($$=($_=!''+$)[_/_]+$_[+$])])()[__[_/_]+__
[_+~$]+$_[_]+$$](_/_)

Yep, perfectly legal Javascript, Care to hazzard a guess as it’s function?

http://adamcecc.blogspot.com/2011/01/javascript.html

^ go here to find out more… It’s fantastic :)

Amazing One Line Script

I was stumbling around on the internet this morning and I found this awesome site… (http://www.centerkey.com/tree/)

This guy posted an amazing one line script that will graphically show your directory tree with a simple one line command:

ls -R | grep ":$" | sed -e 's/:$//' -e 's/[^-][^\/]*\//--/g' -e 's/^/ /' -e 's/-/|/'

and that’s all!!! It works great!

MAME cabinet, on Debian (Part 1)

Okay, so generally you know I’ve been up to no good (contracts) when I haevn’t posted in some time. It’s true, I’ve been working pretty hard lately.

So last night I sat while watching the olympics it made me think of Hyper Olympic, the name of the console that I got for my Mame Cabinet.

I decided to finally install an OS on the machine I’ve devoted to my arcade console. Since this is just first step, I don’t have pictures for you yet – but I will – rest assured.

So I started with a clean debian netinst cd for Debian 5.0 (Lenny).

I started by installing the “Advanced install“.  I want my stystem to be as slim as possible, so I want to control as many options as I can while installing.

After the usual language, keybaord layout and detection of the CDROM and installation media the fun started.

I set my hostname to “Hyperolympic”; the name of the game my cabnet came from.  Later on I hope that it will default to “hyper olympic” as a screensaver – but right now, it’s just the host name.

I also configured it to allow me to contiue working on it from SSH (which allowed me to watch the USA team whoop Canada’s Hocky team), from there I configured it to use the time.nrc.ca atomic clock for ntp, and start to parition the disks up.

The drive I have in it is an old 30Gb hard disk, the ROMS I’ll store on my NAS so It only really needs to hold temporary data and so the drive doesn’t need to be very large.  I used an old 2Gb USB Thumbdrive as a “swap” partition.  Very simple partition scheme, the 30GB drive is a the root (/) paritition, the entire USB Thumb Drive is swap.

Install the base system, and cheer on the Canadians.  I created a user called “interface” who will be the “interface” user, I disabled root here as well and do all interactive root sessions by using “sudo -i” from my interface user.  I’ve seen a few tutorials that start the mame arade as root, but I disagree with this practise. 

Finally it asked for some “task specific” kinds of installation options.  I chose “Standard System Only” (or similar), I have no need for apache, dns, or an xwindows desktop environment.

Install grub (not grub2) and ta-da.   A bootable debian.

Next, I logged in as interface, sudo -i to root, and edited my /etc/apt/sources.list.  I found this neat place (http://apt.ludomatic.fr/?hl=en#repo) who has mame sdl, precomplied for debain.  Following their instructions I added the following lines to my sources.list:

## LUDOMATIC REPOSITORY
deb http://apt.ludomatic.fr lenny non-free
deb-src http://apt.ludomatic.fr lenny non-free

Then using a quick command (they also show) I get their security keys:

wget http://apt.ludomatic.fr/ludomatic.key.asc -O - | apt-key add -

I run apt-get update to ensure there are no errors (there weren’t).

Now I have to enable the framebuffer.  There’s not a whole lot if info about this – but it turns out it’s pretty easy.  I added the “vga=791 video=nvidiafb” to my boot.1st file and rebooted.  Obviously this is for nvidia cards, if you don’t know your card you can use the “video=vesafb” instead, or you can use your own… there’s plenty and you can find them all in:

“/lib/modules/”uname -r`/kernel/drivers/video/”

You also should edit your /etc/modules to include your framebuffer drive too.  I also installed “splashy” which allows me to choose themes and “boot up screens”

apt-get install splashy splashy-themes

Once all of that is complete I now install GCC and build utilities so that I can build the “advanced menu” system.

apt-get install gcc g++ binutils make

Then in my /usr/local/src directory I download the advance menu sources:

wget http://prdownloads.sourceforge.net/advancemame/advancemenu-2.5.0.tar.gz?download
tar -vxzf advancemenu*.tar.gz
cd advancemenu-2.5.0
./configure && make install

Now, SDL Mame.

apt-get install sdlmame

Next I edit my /boot/grub/menu.1st and change the “timeout   5” line to “timeout 3” to speed up boot, I also changed the /etc/init.d/rc script the line that reads “CONCURRENCY=none” to “CONCURRENCY=shell“

. . . and Reboot!

Ta-DA :)  A bootable system, not much is going on – you can start messing with the advance menu system, which I’ll cover next time :)

to be continued…

Video Capture Card Driver Woes

So lately I’ve been working on copying my old VHS movies to digital format so that my NAS can serve them up to my XBMC.

In my stumbles around the internet I found this really great project.

It seems to include EVERY older Video Capture Card for Windows drivers all into 1 beautiful fast and easy to install package.

Since I don’t use windows, I don’t really have much to do with it, but it’s good for those of you who do.

Fun

http://btwincap.sourceforge.net/

Take a look

Back from the Dead – Fixing Drives in Linux

So as you all know I lost my NAS due to some disk errors. 

Sadly, it was actually my flash adapter card (http://www.dealextreme.com/details.dx/sku.711) failed and this caused my FreeNAS to fail.

After much debugging – since this solid state drive seemed like a power supply failing: power turns off a couple seconds after being turned on, turning it on sometimes would do nothing until you unplugged the power supply then plug it back in.  (http://hubpages.com/hub/How-to-Tell-When-Computer-Power-Supply-Is-Failing)

Anyway, after figuring it out, I put in an old unlocked X Box hard disk (15 Gb) and that’s now my boot drive.  I used the newest version of FreeNAS which allows for a “full install” on a hard disk. (simply boot it up, and use the console menu to install)

Now that I’ve got my FreeNAS server up again, I still have a broken 500Gb drive to fix.

I took the 500Gb drive out and put it in my Ubuntu Linux lab server.  I usually use this machine for any experiments that I’m doing and for virtual machines.

Canada computers and a few others have some pretty good deals on 1Tb drives right now.   (I got mine for $90.00. http://www.canadacomputers.com/index.php?do=ShowProduct&cmd=pd&pid=024109&cid=HDD.443.877)  So this is a good time to upgrade my drive.

I put it into my lab server, and dd’ed the drive to make a complete copy onto my new 1Tb drive. 

Now I have a whole bitwise copy of my drive, now the fun part.

I don’t know UFS as well as I know FAT or EXT, so I was hoping to find a utility to help me with the restoration of this drive.

A quick google found me a utility called “testdisk”, (http://www.cgsecurity.org/wiki/TestDisk) and since I have a bitwise copy of my drive, I thought let’s give it a try.

Ran it,  it took literally 1 hour to analyze the drive.  It was so slow that I almost gave up.  However, beer and a night of Simpsons let me relax and let it do its work.

I wish I’d taken screen shots for you, but I was thrilled to find out that it actually worked. 

It recognized the UFS filesystem, GFI partition and all.  It rewrote the partition table, and dropped me to a command prompt.

I installed “ufsutils” with apt-get install, and ran fsck.ufs on it.  fsck did it usual “i found xxx fix?” and the drive was repaired.

I copied the data from the old 500Gb to the new 1Tb, and installed the new 1Tb drive into my FreeNAS server, added the drive, and ta-da it worked.

Back in business.

Now, the new version of FreeNAS has some neat new features that I discovered, the least of which is “/etc/rc.d/transmission blocklist-update” – which makes my old tutorial redundant, and I’ll write an update about that sometime soon :)

This week I’m going to reconfigure it – and I’ll probably write a little tutorial for you all who are interested.

In the meantime, I got my NAS back – and my movies.

Revisit Royale Noir

I know I’ve failed to post for the last little while.  But with good reason, my FreeNAS server crashed (destroying the drive) which I intend to post about once I’ve fixed it, I have also been writing a few pieces of software for my clients, and 1 for the open source community (again, I will post this once I have something more “polished” for general consumption)

But, since i’ve been looking at the same page of posts for the last couple of weeks i thought i owe it to you to write something… anything, just to keep you (and me) on our toes.

Since this is something of a revisit but I think it’s quite catching and really a nice personal touch that many of us can use to make our windows boxes just a little more unique.

It’s called “Royale Noir”, a secret hidden XP theme that is only available using a special “luna.msstyles” visual settings file by Microsoft.

Installation is a snap, you just download the royale_noir.zip file that I literally stole from the old “istartedsomething” post (see: http://www.istartedsomething.com/uploads/royale_noir.zip) unzip it into a “royale noir” subdirectory in your “%WINDIR%\resources\themes\” folder, double click on the “luna.msstyles” file and select “Noir” from the colour scheme.

Beautiful.

 

Royal Noir Display Properties

If you want to see the original post: http://www.istartedsomething.com/20061029/royale-noir/

Burn a CDROM to a friend?

Ever wanted to send your friend a cdrom, but didn’t want to send an iso or whatever?  this is for you!

http://0xcc.net/cdfax/index.html.en

Alternative, Alternative Operating Systems

I love operating systems, the heart of any good computer.  Because of my love for the “OS” I have become a sort of collector of Operating Systems.  My virtual machine “LAB” contains nearly one hundred different operating systems. 

Because of this interest I’m going to list a few of the less famous but still noteworthy ones here, sort of top 3 of the unknown OS market.

1. Haiku (http://www.haiku-os.org/)

• I LOVED BeOS, I still do.  It had a very slick interface, beautiful to look at, and very multimedia friendly.  I heard rumors that this was going to be the next Mac OS (before X) but that it didn’t pan out.  Palm bought BeOS, and grounded the OS.  Hooray for open-source, where teams of guys like me (but not me) went out and decided to build a binary compatible version of BeOS.  That’s the origins of Haiku.

2. ReactOS (http://www.reactos.org/)

• I hate Windows.  It was poorly designed from the beginning, because of its single user mentality it just hasn’t been able to catch up.  Maybe this will change in Windows 7 (though I doubt it).  Hooray for open-source, where teams of guys like me (but no, not me) went out and decided to build a FREE binary compatible version of Windows.  That’s te origins of ReactOS, which to me looks fairly useable, and actually quite close to Windows binary compatibility.

3. SkyOS (http://www.skyos.org/)

• SkyOS is an attempt at building a FAST and stable OS from the ground up.  In Assembly.  Quite a project, and it’s quite beautiful.  It has a very quick kernel, and well optimized source.  The problem is that development is very slow, and updates are few and far between.  But quite a nice operating system to watch.

I have personally installed and run each of these with little to no problems on an older machine.  They’re quite nice projects, and (at least in my opinion) nearly user friendly enough for the common desktop market (especially in the cases of Haiku and ReactOS)