ejes consulting

Techincal Consulting Design and Automation

Archive for the ‘Neat Stuff/Good Ideas’ Category

Mac OS X Swap File Optimization

leave a comment »

So I like OS X.  It’s a UNIX desktop done well, it’s well designed, beautiful and easy to use.

With all I love about OS X, there are things that I absolutely HATE.  They could, for example, use the Unix Filesystem Hierarchy (https://en.wikipedia.org/wiki/Filesystem_Hierarchy_Standard) but this article is more about how there is no easy way to modify your swap file and it’s usage.

I understand the desire to keep these kinds of settings under the covers from users who may be in experienced or not know exactly what they’re doing, but they could make it a little simpler to get your hands on.

Nevertheless, it’s not impossible.  During my research, I came upon this guys site (http://osxdaily.com/2010/10/08/mac-virtual-memory-swap/) and with this info (read it, it’s short) you can re-mount your swap files on a new filesystem.

In OS X, swap files are stored at /private/var/vm, so that’s where we’ll concentrate our filesystem changes.

First things first, though, we have to disable the swap files.  Thankfully, this can be done from the command prompt, but before you do disable virtual memory, you should not be running any applications.  If possible disable all network services, and close all programs before releasing all virtual memory, if you leave programs, (especially swapped, unfocused or background ones) open, you may have problems, otherwise consider yourself warned.

Disable “Dynamic Pager” (swap):

Mac-Mini:~ root# launchctl unload -w /System/Library/LaunchDaemons/com.apple.dynamic_pager.plist

After this command is done, you can safely delete everything in the /private/var/vm/ directory.

Mac-Mini:~ root# rm /private/var/vm/*

Then simply mount your new filesystem.  Mine is “msdos”, I figured that having less overhead would make it slightly quicker.   Though a security remember: these files would be readable by perhaps the wrong people, and could be used for malicious reasons or for information gathering purposes (ex. http://www.linuxjournal.com/content/doing-reverse-hex-dump).

Mac-Mini:~ root# mount -t msdos /dev/disk1s1 /private/var/vm

Then you can re-enable dynamic pager (swap, virtual memory), don’t forget the “-wF” the F probably means “force”:

Mac-Mini:~ root# launchctl load -wF /System/Library/LaunchDaemons/com.apple.dynamic_pager.plist

There you have it, your swap file will grow on this partition.

Please remember that you have to be root to run these, if you are unable to become root, please consult the manual pages for “sudo” on OS X.

Advertisements

Written by ejes

December 28, 2013 at 10:58 pm

pSearch Source!!

leave a comment »

So I had some trouble getting my source put onto wordpress.  I can understand their point, they don’t want to share .zip, .tar or any other archive container formats.

In the intrest of brevity, I decided to just use a free file host.  I chose medifire, it was top on google when I checked.

http://www.mediafire.com/?u01bf1nwbemata9

There is where you can find the historical archives of my search development.  I did my best to ensure that it could be compiled on Windows (32-bit XP via, MinGW) or Linux (ubuntu 64-bit server), sometimes OpenBSD. You’ll likely need sqlite (http://www.sqlite.org/) and libcurl (http://curl.haxx.se/), you’ll probably need pcre libraries as well (http://www.pcre.org/).  If you try to compile something that looks like it should work, let me know and I’ll see if there’s any libraries that I might be missing, or at least I can let you know if it SHOULD compile.

All the above source is released under the original BSD Licence.

Copyright (c) 2010-, Evan Stawnyczy (ejes consulting) ejes@torfree.net
All rights reserved.

Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are met:
1. Redistributions of source code must retain the above copyright
   notice, this list of conditions and the following disclaimer.
2. Redistributions in binary form must reproduce the above copyright
   notice, this list of conditions and the following disclaimer in the
   documentation and/or other materials provided with the distribution.
3. All advertising materials mentioning features or use of this software
   must display the following acknowledgement:
   This product includes software developed by ejes consulting.
4. Neither the names ejes consulting, Evan Stawnyczy nor the
   names of its addional contributors may be used to endorse or promote
   products derived from this software without specific prior written
   permission.

THIS SOFTWARE IS PROVIDED BY EVAN STAWNYCZY AND EJES CONSULTING ''AS IS''
AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
ARE DISCLAIMED. IN NO EVENT SHALL <COPYRIGHT HOLDER> BE LIABLE FOR ANY
DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

As of right now, I’m working on a complete rewrite.

This rewrite I’m hoping will serve as good working reference code.  It will be clear, easy to understand and most likely SLOW.  I am now using ODBC instead of limiting users to sqlite.  This also will hopefully allow enterprises to adopt use without too much trouble.  I decided to scrap the web-server aspect – hopefully someone will want to bundle search and web-server for home use.  Especially with the wide adoption of ipv6 a single workstation could easily share information with others all indexed through a private search network.  At the very least someone should write an nginx (http://nginx.org/), lighttpd (http://www.lighttpd.net/) and apache (http://www.apache.org/) module that indexes static and cached content and publishes the search results.

That leads me to trying to build this as an ip-agnostic application. I want it to run in both an ipv4 and ipv6 network. Of course this has it’s own challenges as well.  I’m trying to maintain ANSI compliance where I possibly can so that it can be easily portable, and mostly so that it can run on windows or unix without too much trouble.

Written by ejes

November 11, 2011 at 2:54 pm

pSearch – a peer to peer, distributed search engine

leave a comment »

Forward

So I haven’t been posting very much for the last while, and this is mainly because I’ve been very busy.

I always have several projects on the go, and I don’t have enough time to devote to all of these things at once, so usually the least interesting project gets placed on the back burner.

That is what happened to this blog.

Now I’ve spent a great deal of time on this, and have produced some very good design documents as well as a bunch of source code.  So… Without further ado

This is my distributed, peer-to-peer search engine.

Attached to this post you’ll find a couple of architecture documents, a pdf with a visual diagram of how this engine is suppose to work, and another pdf with a long winded, half written description of why and how I expect this conceptually to run.

I’m not a writer, and am mostly a technical person, however, I am actively updating and modifying this project so expect updates as it goes.

The first document is the “pSearch – Document

In this document I attempt to explain the strategy, and reasons for this project and what  I hope that it will accomplish.   This document is incomplete, but I encourage you to read it anyway.

The second document is the “pSearch – Drawing

In this document I have detailed the major aspects of the distributed search.  Hopefully it’s easy to follow, I don’t expect this diagram to change very much.

And I have a LOT of source code that I still have to organize – much of it will be posted here and some of it is too embarrassing.

Summary

So, without drudging into my documentation in too much detail (I posted them above, feel free) a simplified “how does this work” seems appropriate.

Each peer will accept connections  from the internet.  Each search request is forwarded to other peers as defined in it’s database.

While this happens, it also uses a second task to search it’s own internal database.  On a private home machine this internal crawler has a small collection of sites and keywords based on several configurable data collection points (such as your browser cache, or installed programs) which would automatically include a lot of data that would be specific to you.   A public internet site would index their own pages (this isn’t mandatory, but preferred).

After that, it’s a simple case of matching the keyword and publishing the results to the connected client.

Peers who respond quickly, and with a lot of results are flagged as “experts” when it comes to this set of terms.  This way, when you search for a similar set of terms again, the “expert” peers will be consulted first.

This way, common search terms will be responded to by clients who have a lot of information on these terms.  For example a site that indexes movies (like imdb) would respond with a lot of results for movie titles and information about films, but probably have very little to respond when a query has some specific request about cars.

Expect more as I develop more.  I encourage anyone to read and comment about my designs.

Written by ejes

November 7, 2011 at 11:43 am

Javascript Obfuscation to the MAX!!!

with one comment

I was stumbling around the internet today and found this:

($=[$=[]][(__=!$+$)[_=-~-~-~$]+({}+$)[_/_]+
($$=($_=!''+$)[_/_]+$_[+$])])()[__[_/_]+__
[_+~$]+$_[_]+$$](_/_)

Yep, perfectly legal Javascript, Care to hazzard a guess as it’s function?

http://adamcecc.blogspot.com/2011/01/javascript.html

^ go here to find out more… It’s fantastic :)

Written by ejes

February 1, 2011 at 2:47 pm

Amazing One Line Script

leave a comment »

I was stumbling around on the internet this morning and I found this awesome site… (http://www.centerkey.com/tree/)

This guy posted an amazing one line script that will graphically show your directory tree with a simple one line command:

ls -R | grep ":$" | sed -e 's/:$//' -e 's/[^-][^\/]*\//--/g' -e 's/^/ /' -e 's/-/|/'

and that’s all!!! It works great!

Written by ejes

March 17, 2010 at 11:42 am

MAME cabinet, on Debian (Part 1)

with 6 comments

Okay, so generally you know I’ve been up to no good (contracts) when I haevn’t posted in some time. It’s true, I’ve been working pretty hard lately.

So last night I sat while watching the olympics it made me think of Hyper Olympic, the name of the console that I got for my Mame Cabinet.

I decided to finally install an OS on the machine I’ve devoted to my arcade console. Since this is just first step, I don’t have pictures for you yet – but I will – rest assured.

So I started with a clean debian netinst cd for Debian 5.0 (Lenny).

I started by installing the “Advanced install“.  I want my stystem to be as slim as possible, so I want to control as many options as I can while installing.

After the usual language, keybaord layout and detection of the CDROM and installation media the fun started.

I set my hostname to “Hyperolympic”; the name of the game my cabnet came from.  Later on I hope that it will default to “hyper olympic” as a screensaver – but right now, it’s just the host name.

I also configured it to allow me to contiue working on it from SSH (which allowed me to watch the USA team whoop Canada’s Hocky team), from there I configured it to use the time.nrc.ca atomic clock for ntp, and start to parition the disks up.

The drive I have in it is an old 30Gb hard disk, the ROMS I’ll store on my NAS so It only really needs to hold temporary data and so the drive doesn’t need to be very large.  I used an old 2Gb USB Thumbdrive as a “swap” partition.  Very simple partition scheme, the 30GB drive is a the root (/) paritition, the entire USB Thumb Drive is swap.

Install the base system, and cheer on the Canadians.  I created a user called “interface” who will be the “interface” user, I disabled root here as well and do all interactive root sessions by using “sudo -i” from my interface user.  I’ve seen a few tutorials that start the mame arade as root, but I disagree with this practise. 

Finally it asked for some “task specific” kinds of installation options.  I chose “Standard System Only” (or similar), I have no need for apache, dns, or an xwindows desktop environment.

Install grub (not grub2) and ta-da.   A bootable debian.

Next, I logged in as interface, sudo -i to root, and edited my /etc/apt/sources.list.  I found this neat place (http://apt.ludomatic.fr/?hl=en#repo) who has mame sdl, precomplied for debain.  Following their instructions I added the following lines to my sources.list:

## LUDOMATIC REPOSITORY
deb http://apt.ludomatic.fr lenny non-free
deb-src http://apt.ludomatic.fr lenny non-free

Then using a quick command (they also show) I get their security keys:

wget http://apt.ludomatic.fr/ludomatic.key.asc -O - | apt-key add -

I run apt-get update to ensure there are no errors (there weren’t).

Now I have to enable the framebuffer.  There’s not a whole lot if info about this – but it turns out it’s pretty easy.  I added the “vga=791 video=nvidiafb” to my boot.1st file and rebooted.  Obviously this is for nvidia cards, if you don’t know your card you can use the “video=vesafb” instead, or you can use your own… there’s plenty and you can find them all in:

“/lib/modules/”uname -r`/kernel/drivers/video/”

You also should edit your /etc/modules to include your framebuffer drive too.  I also installed “splashy” which allows me to choose themes and “boot up screens”

apt-get install splashy splashy-themes

Once all of that is complete I now install GCC and build utilities so that I can build the “advanced menu” system.

apt-get install gcc g++ binutils make

Then in my /usr/local/src directory I download the advance menu sources:

wget http://prdownloads.sourceforge.net/advancemame/advancemenu-2.5.0.tar.gz?download
tar -vxzf advancemenu*.tar.gz
cd advancemenu-2.5.0
./configure && make install

Now, SDL Mame.

apt-get install sdlmame

Next I edit my /boot/grub/menu.1st and change the “timeout   5” line to “timeout 3” to speed up boot, I also changed the /etc/init.d/rc script the line that reads “CONCURRENCY=none” to “CONCURRENCY=shell

. . . and Reboot!

Ta-DA :)  A bootable system, not much is going on – you can start messing with the advance menu system, which I’ll cover next time :)

to be continued…

Written by ejes

February 23, 2010 at 11:00 am

Video Capture Card Driver Woes

leave a comment »

So lately I’ve been working on copying my old VHS movies to digital format so that my NAS can serve them up to my XBMC.

In my stumbles around the internet I found this really great project.

It seems to include EVERY older Video Capture Card for Windows drivers all into 1 beautiful fast and easy to install package.

Since I don’t use windows, I don’t really have much to do with it, but it’s good for those of you who do.

Fun
http://btwincap.sourceforge.net/
Take a look

Written by ejes

November 4, 2009 at 3:29 pm