Category Archives: Technical

Base Conversions in MySQL

Allow me to present dec2base() and base2dec(), a pair of MySQL stored procedures for converting to/from denary and other number bases. The examples below are for Base 53. Just add or remove characters from t_digits to change.

If you are curious about my choice of Base 53, it is because I have tried to minimise the use of similar-looking characters, to reduce transcription errors.

This code is based on an example in PL/SQL that I found on the JavaConfessions blog.

The reason for using varbinary() instead of varchar() for t_digits is that base2dec() will produce incorrect results due to case-insensitive matches on varchar().


drop procedure if exists dec2base;
delimiter //
create procedure dec2base(n int unsigned)
begin
declare t_modulo int unsigned;
declare t_int int unsigned;
declare t_val varchar(256);
declare t_char char(1);
declare t_digits varbinary(256);
declare t_base int unsigned;

set t_digits='23456789abcdefghjkmnpqrstuvwxyABCDEFGHJKLMNPQRSTUVWXY';
set t_base=length(t_digits);

if (n=0) then
select 0;
else
set t_int=n;
set t_val='';

theLoop: loop
if (t_int=0) then
leave theLoop;
end if;
set t_modulo = t_int % t_base;
set t_char = substr(t_digits, t_modulo+1, 1);
set t_val = concat(t_char,t_val);
set t_int = floor(t_int/t_base);
end loop;

select t_val as base_value;

end if;
end;
//
delimiter ;

drop procedure if exists base2dec;
delimiter //
create procedure base2dec(c varbinary(256))
begin
declare t_iterator int unsigned;
declare t_length int unsigned;
declare t_char char(1);
declare t_int int unsigned;
declare t_retval int unsigned default 0;
declare t_mult int unsigned default 1;
declare t_convval varbinary(256);
declare t_digits varbinary(256);
declare t_base int unsigned;

set t_digits='23456789abcdefghjkmnpqrstuvwxyABCDEFGHJKLMNPQRSTUVWXY';
set t_base=length(t_digits);

set t_convval=c;
set t_length=length(t_convval);
set t_iterator=t_length;

theLoop: loop
if (t_iterator=0) then
leave theLoop;
end if;
set t_char = substr(t_convval,t_iterator,1);
set t_int = instr(t_digits,t_char)-1;
set t_retval = t_retval + (t_int * t_mult);
set t_mult = t_mult * t_base;
set t_iterator = t_iterator -1;
end loop;

select t_retval as decimal_value;

end;
//
delimiter ;

Vodafone Australia Mobile Broadband – On Linux

I have just acquired a Huawei K3250 USB wireless modem with a Vodafone prepaid SIM.  There seem to be a lot of "recipes" floating around on how to get similar devices working on Linux, but several which supposedly work with the service from Vodafone Australia, in a word, don't.

My laptop is running Debian Lenny with a 2.6.26 kernel – recent kernel versions supporting the Huawei devices by default.

As I'm a bit old-fashioned and always used wvdial back in my dial-up days, this is what I have used this time around too.

This is the wvdial.conf that works for me:


[Dialer Defaults]
New PPPD = yes
Stupid Mode = 1
Init1 = AT
Init2 = ATZ
Init3 = ATQ0 V1 E1 S0=0 &C1 &D2 +FCLASS=0
Init4 = AT+CGDCONT=1,"IP","vfinternet.au"
Modem Type = Analog Modem
ISDN = 0
Phone = *99***2#
Modem = /dev/ttyUSB0
username = vodafone
Password = vodafone
Dial Command = ATDT
Baud =460800

Issues that may be encountered include:

  • The user in question not being in the dialout group
  • /usr/sbin/pppd having the incorrect permissions (should be -rwsr-xr–)
  • Using one of the many example wvdial.conf's that gives a phone number of *99#. This was the last problem I encountered – I would get a connection and then the modem would get fed up and stop pppd. Only when I changed to *99***2# did things start working as they should.

My only complaint is that it ain't half slow after 1.5Mbps ADSL – but I've only got it to keep me working during ADSL outages and for the very rare occasions when I am not at home.

Why did I choose Vodafone? Well, my first choice would have been my excellent ISP, Internode, but their service is through a carrier that doesn't have coverage in this area. The only other carrier with service around here is Telstra/Bigpond – who demand an arm, both legs, and your first-born for the privilege of connecting to their network. They will also add insult to injury with what I would laughingly call their customer service.

And when it comes to service, Vodafone Australia may give itself a good pat on the back both for their sales staff who got my order through despite having a rural South Australian address (impossible online,) and for the chap who helped me out this morning getting the thing connected.

In and Out of SVG

SVG

Preamble

I have been looking at SVG for some time now. Why? Because it is an open (non-proprietary) format and because it is a W3C recommendation. I like to use open standards as much as possible in my work (and would like everyone else to do the same) so SVG has always looked attractive compared to the likes of Adobe Flash and the emerging <canvas> element of HTML5. Best of all, SVG has inherent the means to be made accessible.

SVG forms the core of some visualisation tools that I am producing for a client for their Tribal Tool-Kit. The work in which I am engaged at the moment has raised some issues. This article describes those issues and how I am addressing them and constitutes "stuff I wish I had known when I started" as well as an aide-memoire for the next time I come across them. Hopefully others will benefit from this and not have to spend quite so many hours staring at Google as I did.

SVG Not The End Format

Whilst I would love to be able to render data as SVG and leave it at that, there are reasons why this cannot always be so. There's a little graphic on this page that says "SVG". It's not an SVG file though, it's a PNG. (I created it as SVG – the original is here.) Why is this? Because Internet Explorer, almost alone of modern web browsers, does not support SVG. Developers Brad Neuberg (Google Inc.) et al are working on svgweb, in an attempt to rectify this situation. However, as svgweb is only in the alpha stage of development and because I am wary of being an early-adopter (especially when my clients are concerned,) I will continue to convert SVG for the web to PNG.

My current project has PDF as an end-format.  Rather conveniently, SVG can be converted to both PNG and PDF using the same tool, rasterizer, which comes as part of the Apache Batik toolkit.  I should note at this point that the export tools that I am discussing need to run on a web server; they are not desktop applications. (They can, however, be run from a command prompt. I work on Linux, but the majority of the tools that I am discussing are available cross-platform.)

Converting an SVG file to PNG is as simple as installing Batik and running:

rasterizer foo.svg

…to produce foo.png. To convert to PDF, we need to do a bit more typing:

rasterizer -m application/pdf foo.svg

…to produce foo.pdf.

Easy, isn't it?  The current project, however, produces 7 SVG files but requires 1 PDF document as the end product.  I thought that this might complicate matters but thanks to a nice little application called pdftk, all we need to do is to convert the 7 SVG files to PDF (I'll call these doc1.pdf … doc7.pdf) and then run:

pdftk doc1.pdf doc2.pdf doc3.pdf doc4.pdf doc5.pdf doc6.pdf doc7.pdf cat output mydoc.pdf

…where mydoc.pdf is a concatenation of doc1.pdf … doc7.pdf.

Caveat

This may sound easy – and it is. The caveat, however, is the fact that it isn't necessarily quick.  If you are rasterizing SVG on a a web server, assume that this process will have to run asynchronously and notify the end user that they may have to wait a bit.

Oh, and if anyone feels that there is something wrong about creating PDFs with a tool called 'rasterizer' when PDF isn't a raster format, yes, it's weird. Ignore it; it's just a name.

The Branding Problem

The PDF document that I am creating from that bunch of SVG files is a report. It needs to be branded with logos. Fair enough. You just go to the graphic designer that created the logos and ask for them in SVG format. Possible, if they were created with the likes of Adobe Illustrator. More likely, this is not the case.  Odds are that you will end up with a raster format such as TIFF or – if you are especially lucky – with encapsulated PostScript (EPS.)  Let's deal with the raster formats first.

We can put an external image into our SVG easily enough:

<image xlink:href="http://www.smiffysplace.com/img/png.svg"/>

If we were odd enough to want to put a rasterization of an SVG file into another SVG file, that is.

When rendered, this all looks OK – until we try zooming in on the image. Those parts that are pure SVG will scale and scale and scale – and still be as sharp as at 1:1 magnification. The raster graphics, however, will become horribly pixellated and very 8-bit looking. (That's a joke for those who used to have Sinclair, Commodore, Atari and the like home computers.)

If there is no choice but to use raster graphics, the trick is to start with an image larger than you require and scale it down using the width and height attributes of the image element.  Doing this means that you should not start to see pixelisation until you have zoomed past the original size of the raster graphic.  For example, if the raster graphic is 200×200 pixels and we size it to 50×50 pixels in the SVG, it should look fine until we magnify the SVG beyond 400%.

But we're lucky enough to have EPS. Why's that lucky? What's so great about PostScript? What's good here is that we can convert to SVG without loss. Once we have our logo as SVG, we can just pop it in the <defs></defs> section of our SVG, job done.

To convert EPS to SVG, the first thing we need to do is to download and install the Open Source Desktop Publishing  (DTP) package, Scribus. Then:

  1. Start Scribus.
  2. When a dialogue pops up, select the tab 'Open Existing Document'.
  3. In the file selector, change File type to: PostScript(*.ps *.PS *.eps *.EPS)
  4. Select your EPS file, then OK.
  5. Scribus will prompt you for a file name for its native format. Put in something appropriate or just accept its suggestion.
  6. Remove any unwanted bits in the document now showing.
  7. From the File menu, select Export, then Save Page as SVG.

You will now have an SVG file with your desired logo in it. There's a fair chance that you have a little logo sitting all alone in the corner of a big SVG. If this is the case, change the height, width and viewbox attributes of the SVG element to suit. Tip: if you don't know what size to use, rasterize the SVG, open the resulting PNG in an image editor such as the GIMP, where you will be able to see/measure the part of the image you want.

Update

I thought I had a working TIFF to SVG technique, which I posted here briefly, which used Open Office Draw as a step to creating an EPS. However, whilst this appeared to work, I found that the SVG file created had an embedded PNG of the original image in it! I will continue my research and post any updates here.

Notable Mention

There is one Open Source SVG tool that I did not use for this project but is worth a mention: Inkscape. I appreciate that not everyone likes to create their SVG by hand, as I do, so Inkscape makes a good tool for an introduction to the wonderful world of vector graphics.

Conclusion

There is generally more than one way to do a job. What I have described in this article is how I export from and import to SVG.  Google Is Your Friend – have a look for alternatives and see what you can learn in the process. (Anyone trying to concatenate PDFs using GhostScript should be warned, however, that it trashes embedded raster graphics.)

Have fun with your SVG!

Dear Oracle…

Dear Oracle Management

Allow me to congratulate you on your prospective purchase of Sun Microsystems (regulators willing.)  The potential synergy of your business methods and Sun's products is quite awesome.

Another thing that would be quite awesome would be if you were to allow the use of your database and tools under the same FOSS terms as Solaris, MySQL, Java, Glassfish, and friends.  As a small, independent, developer I would love to integrate your products that have, to date, been outside of the financial reach of both myself and my clients. In doing so, I would be able to help increase your market penetration.

I hope that the acquisition goes smoothly for you.

Sincerely yours

A. Developer

Smiffy’s Random Tech Tips

Preamble

(Listen carefully, I shall say this only once!)

Every now and then I feel a need to communicate various little technical tips that stem from my work or stuff that I observe on the web.  They rarely get written up because they tend to be too short to justify an article on Smiffy’s Place and, now that I am using Twitter, too long to fit into a silly 140 character limit, even when written in Newspeak and with ridiculous abbreviations.

Today I decided to start making collections of these as draft articles, publishing those articles when I felt that I had enough to make it worth anyone’s while to read them.

The number of tips per article is something on which I am, as yet, undecided.  Somewhere between 3 and 10 sounds sensible to me, so I may well just stick at that.

The First Tips

(X)HTML authoring – the Joy of Comments

Add comments to the end of block-level elements with IDs so that you can see what you are closing and pick up any orphaned </div>s, etcetera, that are stopping your code from validating.  (This can also apply to classed elements.)

This can also help other people working on the same project keep track of what you are doing, say you are writing backend code and someone else is doing the styling.  (Documenting your IDs and classes can save others on the project much frustration and puzzlement.)

Unless you have a specific reason to retain them, comments may be removed easily using regular expressions or other pattern patching. Note that superfluous comments can contribute to the overall weight of a page and thus slow down your site for visitors and also eat into your bandwidth.

Hardware repairs, modifications – Use a Camera

Where hardware can be anything from laptops to washing machines to cars to X-Wing Fighters.

In addition to the tools that you would use regularly, have a digital camera on hand and photograph every stage of disassembly, especially if you do not have a manual.

I recently had cause to disassemble our wheelbarrow so that it could be stripped of its defective powdercoat and painted.  There was a gap of a couple of months between the disassembly and the painting/reassembly.  Without the photographs that I took, I would probably have a very strange-looking wheelbarrow now.

The “use a camera” tip can also extend to excavations.  If you are building or renovating a house (shed, office, bat cave, whatever,) photograph all trenches where pipes and cables run, from several angles.  This can be very helpful in the event of any future excavations, fencing work, etcetera.

Don’t forget to file your images where you can find them again (tagging with Picasa can be useful,) and make sure that they are backed up.

Pressure pumps – the Peril of Snails

This one’s a bit obscure – mainly applies to rural Australia.

If you have an outside electric pump on rainwater (or other) service which has not been run for a while, remove the fan cowl (with power disconnected!) before operation to check for infestation of white snails.  These may or may not cause the pump to jam, but should be removed anyway.  Also check that the pump is primed and any inlet valves are on before starting.  And put the fan cowl back.

JavaScript – JSLint

If you are working with large chunks of JavaScript (ECMAScript,) you may wish to consider validation using JSLint.  You may also wish to consider checking that you are using good programming practices: always declare your variables, comment your code, don’t try coding when you are too drunk, etcetera.

Conclusion

There you go, just a little taster.  More will follow, sometime.

TinyURL for this article: http://tinyurl.com/d3jdqb

follow-u-tron

Automating #followfriday

This little programme allows me to manage a list of people that I follow on Twitter whom I feel that others might also to follow as well. When invoked without any parameters, it will draw a random name from the list and post a #followfriday message to Twitter.

The idea is that I just maintain the list and have this running on a cron job every Friday. Picks are random so the same person may be picked more than once in a row.

Thanks to @elpie for reminding me of what day Friday is in cron-speak!

Creative Commons License
This work is licensed under a Creative Commons Attribution 3.0 Unported License. Use it at your own risk, don’t blame me if it eats your computer, causes a Global Economic Meltdown (too late, someone’s already done that, thanks GWB,) etcetera.

The Code

Several attempts were made to copy and paste the code into this article; TinyMCE stuffed up the formatting every time, eating a load of essential backslashes. Rather than convert the whole thing to HTML entities by hand, I’ve given up and have just posted it in my files section.

Usage

Adding Members

follow-u-tron.pl @user1 @user2 @user3

Deleting Members

follow-u-tron.pl -@user2

Note that you can add and delete in the same command.

Listing Members

follow-u-tron list

Posting to Twitter

follow-u-tron

This may be put into a cron job.

@smiffytech – The Twitter and I

Important Note

As of 15 August, 2009, my Twitter identity changed from @smiffytech to @smiffy. The @smiffytech account still exsists, but with a message referring to the new one. Followers/following not affected.

Preamble

Until fairly recently, my only encounters with online social networking were with the professional networking site, LinkedIn. I have to confess that my impression of the other social networking utilities out there was “kids’ stuff”, “silly” and the like. Now that I have delved into these utilities, I have gained considerable respect for them – in concept if not in execution.

There are two main reasons for me looking into and taking up with various online social networking utilities. These are 1) my clients use them and 2) they are web applications; as a web applications developer, I need to keep current with applications being used at large.

I have decided to take up regular Twitter usage because, even in the short space of time that I have been using it, I have made some valuable and interesting contacts. Despite its limitations, Twitter is a valuable tool and, from a business perspective, only a fool would ignore it.

Twitter

The history and actuality of Twitter are a matter of public record so I feel no desperate need to regurgitate a load of facts in this article.

These are what I consider to be the key points of Twitter:

  • Posts are limited to 140 characters. I know the reasons for this, I don’t like them and I don’t like the restriction. But that’s what Twitter is, I don’t see it changing so one must put up with it.
  • Twitter has a publicly available API so anyone can develop their own add-on, value-adding or whatever software. I consider this to be a very important feature.
  • Threading is limited; whilst one can reply to a “tweet”, the reply appears to be linked only to the parent, not to an entire thread.

Usability/Accessibility

Accessible Twitter icon

Twitter has in common with every social networking utility I have seen so far – the web interface is crap. I’m not knocking the developers here specifically – it still seems that only a minority of those developing for the web have even the faintest clue when it comes to accessibility or web standards. However, by offering an API, Twitter gains itself Brownie Points because this allows others to develop better, more accessible interfaces. Step forward Dennis Lembrée, the creator of Accessible Twitter. (Dennis is @AccessibleTwitr on Twitter.) Whilst this service is only in alpha at the time of writing, the work done so far has been truly excellent.

Followers and Following

Whilst one can sample the random noise that is the Twitter live feed, to get any sense out of this vast flow of information, one needs to “follow” specific users. For me, the first people that I follow are those in my own “tribe” – those that I know in real life – friends, colleagues, associates, etcetera. Others that I follow are those who follow me; I choose these based on a certain set of criteria – more of that later.

So, do I just follow everyone that follows me? Absolutely not! Whilst Twitter is important to me as a network builder, I don’t want to spend excessive time looking at it or suffering from information overload.

My “following the followers” are:

  • If a new follower has too few posts that are of interest to me, I don’t follow them.
  • If a new follower has posts of moderate interest but is posting a vast volume, I don’t follow them. Until such time as I have written my own Twitter-reading code to help me sort through posts, I need to keep the incoming volume down to what I can easily handle manually. Twitter may get me new business, but looking at it doesn’t get my work done.
  • If a new follower is following many, has few followers, has no posts and not completed their profile, I will block them straight away.
  • If a new follower is following many, has few followers, has no posts, has completed their profile, but still has no posts after 24 hours, I will block them unless I have good reason to believe that they are genuine users only just getting started.
  • If a new follower has only made spammy posts, I block them straight away.

It’s a Game!

Er, no, I don’t agree. It seems that there are those that use Twitter as a means of self-gratification, measuring their “success” by their number of followers. I believe that any form of success in social networks is the number of meaningful relationships that one establishes – quality being significant rather than quantity.

But each to their own. I want to build professional networks, others want an ego-trip. Twitter appears to be able to provide both.

Further Reading

 

Cheers all!

Smiffy, aka @smiffy

TinyURL for this post: http://tinyurl.com/clar7f

Ada Lovelace Day – Women in Technology Whom I Admire

It starts thus:

“I will publish a blog post on Tuesday 24th March about a woman in technology whom I admire but only if 1,000 other people will do the same.”

this being from coffee.geek.nz

I am fully aware that the technical sector – whether it be IT, engineering or medicine (which I consider technical) – has always been very much a “boys’ club” and am embarassed to be a part of such a misogynistic industry.

Whilst I admire what Ada did (writing programmes without a computer is no mean feat), I would like on this occasion to recognise 3 women of my own acquaintance. They may not be Big Names, but they are just some of those that I can name whom I consider worthy of respect:

  • Leah MACLEAN. Leah is a technology veteran, having worked for Telstra, Australia’s national telco. Leah now provides technology services and understanding to her clientelle (primarily women in small business) as Working Solo.
  • Liddy NEVILE. Liddy Nevile is an Adjunct Associate Professor in the Advanced Computing Research Centre at La Trobe University. I know Liddy as the drive behind OzeWAI web accessibility conference and a member of the Dublin Core Metadata Initiative.
  • Lynne POPE – head honcho of the Mambo CMS project and another technology veteran. A serious Mover and Shaker in Open Source software development.

There are others that I could mention of my personal acquaintance and of no less merit, all just going about their jobs despite the frequent discrimnation that they face. (They not only have to be good at their jobs but also able to survive being Eaten By Trolls.)

Please, let’s just shed the obnoxious attitudes that make the technical world a hostile one for women and rid ourselves of any stupid ideas of women being less able in this – or any other – field.

And yes, I know that this article is over a month early but, by writing it now, at least I know that it will get written.

Shortened URI for this page for Twitter users and the like: http://tinyurl.com/bqmjzk

Here is a list of Ada Lovelace Day posts from others: http://ada.pint.org.uk/

The Solstice Clock – Part 3

schematic of solstice clock prototype
Solstice Clock Schematic

Solstice Clock Circuit Design

In part 1 and part 2 of this series, I started to describe my concept of a Solstice Clock and some of the design parameters. This post is just a brief note to present the circuit schematic for the clock motor driver and the corresponding circuit board layout.

Minor Changes

I have made a minor change to the circuit as described in the previous article. The circuit now includes an 8-way DIP switch rather than a 4-way one. The reason for this is to provide a means by which the length of the pulses driving the clock motor may be controlled. Too short a pulse and the clock may not tick or – even worse – tick intermittently. Too long a pulse and excessive power is consumed. Whilst I do not see the solenoid or microcontroller getting burned out, a reduction in battery life would occur.

In a production environment where only one type of clock mechanism is being used, it is possible to find a pulse length that works with all members of a batch of mechanisms and set the pulse length in the firmware. For prototypes however, especially when different mechanisms may be used, the ability to change the pulse length without re-programming would appear to be most appropriate.

Board Layout

Solstice Clock Board Layout

The image of the board layout presented here is – theoretically – 1:1 scale. It is on my computer but I have no idea how other web browsers may mangle it. For those in doubt, it has about the same footprint as a box of regular safety matches. Before you say "whoopee, isn't it small?" I would advise that this is just the controller board. Somewhere in the back of the clock we need to fit in a pair of AA cells to power the thing. These may be either in two single holders (might make it easier to fit in) or together in a side-by-side holder.

The Solstice Clock – Part 2

Preamble

In Part 1 of this series, I outlined my concept of a Solstice Clock. Since writing the original article, I have modified a regular quartz clock and have estimated the requirements to drive it.

Simplification

My original drive circuit was rather complex – excessively so. Having now reconsidered the requirements (for instance, removing the facility to set the time via a serial data connection) and calculated the drive requirements of the clock solenoid, I have found that a minimal implementation of a Solstice Clock could consist of no more than a microcontroller, timing components a couple of BAT54S dual diodes (protection for the 2 port pins connected to the solenoid) and that's about it.

This more minimal system would use less power (runs on a pair of AA cells – the original would have required 4 or 5) and also fit into far less space than the original. The latter consideration could be important if there is little space in the back of the clock for modifications.

Choosing a Chip

Whilst the very minimal version could be implemented with an Atmel ATTiny45 – making for a very small circuit board – I have decided to make my prototype with a little more flexibility which requires more than the 8 pins of the ATTiny45 and have thus selected the ATTiny2313V. The 'V' is significant – this lower-powered version of the device can operate right down to 1.8V with a clock frequency of less than 4MHz or 2.7V with clock frequencies all the way up to 10MHz.

After playing around with a spreadsheet, I checked some crystal values to see what gives the most accurate results. Actually, it wasn't quite that way – I first looked on eBay for cheap job-lots of crystals in the 3Mhz to <5Mhz range and then ran those through the spreadsheet to find the ones most suitable. I now have batches of 4.9152MHz and 3.579545MHz crystals to play with. There is a trade-off between the time resolution that I can obtain and power consumption; the higher the clock frequency, the better the resolution, but the higher the power drain.

More than just the Tropical Year

My original plan, when I was thinking of controlling the clock through a serial interface, was to be able to vary the 'tick' rate to allow for differing year lengths. Whilst I am now using a mean Tropical Year (so a fixed length), I realised that it would take little effort to provide other mean periods that the clock could operate on. The final list is this:

  • 1 Tropical Year
  • 1 Synodic (Lunar) Month
  • 12 Synodic Months
  • 13 Synodic Months

These 4 periods would be selectable by two positions of a 4-bit DIP switch. This leaves me with 1 switch to control stop/run and another switch reserved for future use.

Setting Up

I have discovered – somewhat to my annoyance – that most quartz clocks provide a means of setting the hour and minute hands, but not one for setting the position of the second hand. As this hand is representing far more substantial periods of time in my slowed-down clock, I have decided to provide the means of setting this hand through the electronics. The stop/run switch will be set to stop, then a push-button held down until the second hand is at the appropriate position. The hour and minute hands would then be adjusted by the regular twiddler on the back and the stop/run switch set to run when ready.

Moving On

In the next article of this series are presented the circuit schematic and board layout of the Solstice Clock prototype.