Server monitoring

Filed under: review,tinkering — jaydublu @ 5:39 pm

mysql_queries-week MuninI think I’ve finally found an almost perfect suite of tools to monitor webserver performance and availability – it’s only taken five years!

The most recent discovery that has me all excited is Munin – I’d heard of it before but can’t think why I’ve never given it a go. It’s a fantastic tool for recording all sorts of useful metrics in rrdtool stylee graphs – far too much info in fact as it’s bringing out my hypochondriac tendencies.

I’ve been using Nagios for years – although Ubuntu distros make it easier to set up it is still a bit like hard work, but once it’s set up it’s great. I’m using nrpe plugins to remotely monitor many of the same metrics as Munin is recording on a suite of servers, but Nagios is set to generate alerts if they go out of tolerance. Once you get the thresholds right it can warn you of impending trouble before a site actually fails – a theory which actually worked a few weeks back when alerts for page response time and processor load allowed me to take evasive action before a site actually crashed.

I’ve got a utility script or two, such as one which monitors MySQL replication, which is regularly polled by Nagios which triggers an alert if a certain string isn’t found. I’m sure there is a plugin or other cunning way to get Nagios to do this without a script, but this was easy, and it works!

Finally for in-house tools, good old AWStats for logfile analysis gives me an idea of raw traffic served.

For remote tools, I use an email to sms gateway to allow Nagios to alert me of critical problems if I’m not at my machine, for a second opinion and as a safeguard I also subscribe to a remote monitoring service – of the many I’ve tried I favour Alertra, but also use Pingdom occasionally. Finally Google Analytics allows traffic analysis within the site, and that’s about it.

But as the BBC says, other services are also available.

Dell support

Filed under: opinion,review — jaydublu @ 6:50 pm

I couldn’t go a whole month without posting something …

My run of technology troubles has continued, and I’m no greater fan of silicon based lifeforms than before.

But I have come to appreciate good support when I get it even more , and I would like to take this opportunity to praise Dell, who are obviously one of these companies who know how much value they can get by looking after their punters once they’ve taken their money.

I turned into a right twonk a few days ago and panicked that I’d screwed my laptop up – terrified that by fiddling I’d make life worse I resorted to contacting Dell support. I won’t embarrass myself by saying what the fix was, but suffice it to say that the very helpful scot on the other end quickly and efficiently put me back on the right track. I’ve rated Dell for years, and this experience has made me even more of a fan.

I can’t write about support without mentioning the other company that I have nothing but praise for and that’s Rackspace – I’ve been at the sharp end of servers playing around previously (with another hosting company), and I have to say that when I’ve been shopping for managed hosting since the first thing I do is check out the support, and the Rackspace claim that their support is fanatical is no understatement – I can vouch for it.

Windows Vista

Filed under: opinion,review — jaydublu @ 12:47 pm

It must be coming on six years ago that I took a deep breath and fired up my first Windows XP machine You see, I’m from Norfolk and we fear change.

I’ve probably used every flavour of Windows way back to Windows 3.0 which went onto good old 286 machines with a couple of floppy disks. I still have a couple of machines that run 98 and Me – they’re doing what they need to and wouldn’t support anything higher, so why bother upgrading?

I’ve always thought Windows a bit frivolous, but as the power of modern machines increases exponentially, I think even these spangly 3d transparent gui interfaces have a hard job using up all available resource, and being one who uses a machine for far too much of the working day, I wouldn’t be without it now. I’m not a good enough typist to survive with a command line interface alone, but I like to think I still could if I needed to (I’m proud to say I use vi regularly!)

I won’t get into the Windows vs. Mac debate, or even Windows vs. Linux – suffice it to say I’ve spent time on most modern desktop environments, and given the choice I still revert to Windows. Servers, well that’s a different matter…

Back to the main story – It was about time to get myself a new working environment, so once again I took a deep breath, and this time volunteered to move into modern times and chose a machine with Windows Vista Business installed.

For reference I’ve been a big fan of XP since that day six years ago – although I will point out that the first thing I do is switch the theme to ‘classic’. I’ve had my share of blue screens, but on the whole I’ve found it as bearable as you could hope for, and I’ve never really hankered for anything else.

So I’ve been using Vista for a few days now… and I’m pleased to report that I’ve had no major issues. I’ve not had to hit F1 yet or otherwise ask for help – everything seems fairly familiar and intuitive. I think I’m a fan of the new interface to Windows Explorer, which from recollection of how OS-X does it seems more like Finder than the old XP Explorer – a crumbtrail instead of an address bar allows you to jump back several levels at a single click.

I’m currently finding it a bit hard to navigate around networks though – ‘My Network Places’ seems to have subtly chaged and it’s catching me out. Similarly, ‘My Documents’ has morphed into just ‘Documents’ – these are just little navigation niggles I’m sure I’ll adjust to as time progresses.

So far I’ve stuck with the default Aero theme, and I’m faintly amused by the ‘glass’ effect, but might try switching back to classic just to be stubborn. And I’m not overly bitten by the widget bug yet, although I do like the big analogue clock on the Desktop.

Here’s hoping I end up as happy with Vista as I have been with XP – I suppose I can always downgrade if I have to.

Gradwell VoIP

Filed under: review,tinkering — jaydublu @ 11:17 am

To open up another can of technical worms, I’ve been investigating voice over IP – I’m about to get into some remote collaborative development so am setting up my tools. I also need a new phone number so I can keep work from home, but being a skinflint I want to do it as cost effectively as I can.

So I read up on VoIP, and investigate Skype and the like, but feel I need something a bit more commercial. Gradwell keeps cropping up as a name, and I’ve had dealings with them in the past and know them to be a reputable company on the bleeding edge of technology.

Grandstream GXP2000So I sign up for a trial, and to do it properly buy a proper phone off them (the Grandstream GXP2000 for reference) but while waiting have a quick play with soft phones – on this case X-Lite.

Now I immediately run into problems that registration with the SIP server is flakey, and although when registered you can ring out or in it’s not often you can get voice through – it’s sounding very much like problems I had when trying to set up videoconferencing, and that turned out to be firewall issues and was only really resolved by putting the conf units on dedicated public IP addresses.

When the Grandstream turned up from Gradwell though, it was preconfigured for my account – and I have to say that I just plugged it in and it worked … almost. There was still something going on in my network that caused connectivity to drop occasionally, which was bad if it happened in the middle of a conversation, but I did have a one hour call with little or no sign that it wasn’t using a ‘proper’ phone. So I’ll put that down to my network and not the service. I’ve since swapped out my router, changed an old HP network hub for a shiny Netgear gigabit switch, and reviewed what boxes are plugged in and chattering away all the time. I’ve also unplugged a redundant wifi access point. Things now seem a lot more stable.

Following on from that long call I decided to extend the trial, and upgraded from a single user account to Gradwell’s ‘Centrex’ package – it basically allows multiple ‘extensions’ to connect as if on a virtual PBX exchange – you can configure all sorts of neat stuff like how external numbers ring through to extensions – IP or external, hunt groups, voice menus, forwarding … and the individual extensions can be anywhere. Very cool.

When I get an opportunity I’m also keen to try using mobile phones that have SIP functionality – my N95 should be one of them but unfortunately I’m with Vodafone who in their infinite wisdom decided that I should be protected from IP telephony and removed those bits of the firmware. Time to swap provider perhaps?

I do have reservations though; although the bandwidth of individual calls isn’t great (less than 100kbps per conversation) the experience is severely compromised if your network isn’t as tight as a gnat’s chuff – anything blocking connectivity will play hell with things. VoIP isn’t the only thing that will suffer, but if the service is business critical it could become a big issue. And that leads me to general concerns about networks – I don’t know if I’m still missing a trick but they are a bugger to diagnose when stuff starts misbehaving. At the moment I think VoIP isn’t for the faint hearted!

It’s still early days and the system hasn’t really had an opportunity to be used in anger, but I’m sure once the initial teething troubles are sorted it will be a fantastic system, with some very useful features and very economical to operate.

Norfolk Enchants – Nimmo Twins

Filed under: review — jaydublu @ 1:38 pm

Nimmo Twins - Normal for Norfolk“Feeling Lucky, Buh?….You’ve got Norfolk Enchants!”

I was at the first night of the rescheduled run of the Nimmo Twins “Normal for Norfolk 11: Bat out of Hellesdon” at the Norwich Playhouse last night, and it was worth the wait. Good to see Karl’s humour (hoomur) seems to have survived his ‘incident‘.

As ever, old characters and content and new (I was also at one of the previews last April), but all righteously hilarious, especially for those who know and love Norfolk. Not for ‘foreigners’ from Ipswich though, or those easily offended.

From a song about the Puppet Man to the tune of Elton John’s Rocket Man, to a speech on behalf of the Mardy Party, to the City Councillors and of course everyone’s favourite She-gew.

It would seem the theme this year is the word ‘mardy‘, and an ongoing fascination with ‘taking it up the wrong’un

Would I go again? “umfug gnu!

Learning to fly the PIC24

Filed under: review,trundle — jaydublu @ 9:06 pm

Programming 16-bit Microcontrollers in CUpdate on Trundle – I’ve been buying everyone else’s Christmas presents for me – a couple of VTX-40 motor controllers, a joystick, a Microchip Explorer 16 evaluation kits and a couple of books on Microcontroller development.

In another life I did a fair bit of PIC programming – on 8 bit devices – but I have to admit I’ve probably forgotten most of it, I worked in assembler, and I know little about the 16 bit RISC architecture let alone programming it in C.

So I do what I normally do when I’m breaking new territory – buy a good book or two.

I’ll own up now that I’ve only read the back cover, the introduction and the first chapter (the obligatory hello world!) but I can already tell I’m going to like it – it’s speaking my language.

Lucio is a private pilot, and the book’s subtitle relates to the way he describes the process of developing in aviation terms; and I think it works well – flight plan, preflight checklist, the flight, post flight briefing and references.

The topics cover are mostly directly relevant to me too – from initial familiarisation with the various environments, references and techniques, through key topics such as I/O, interrupts, A/D, Async comms and for good measure using the onboard LCD display.

Yes it’s going to be a real drag stopping myself from jumping into the fun of later topics, but I know from previous experience that time spent in fundamentals pay dividends down the line.

iPod Touch

Filed under: mobile,review — jaydublu @ 6:31 pm

I finally got my hands on the iPod Touch I got the team for R&D – cheaper than an iPhone – and have spent a bit of time with it myself.

First thing I should say is that dissapointingly I find I really, really like it – it’s a stunning device that’s hard to fault; at least inital impressions are that way.

It’s a perfect size and weight, screen is clear and bright, and as for the ‘touch’ user interface – well what can I say other than ‘wow!‘?

I’ve played a bit with the music and video playing capabilities, and they’re just stunning, but most of my attention has been, unsurprising, on the browser.

Despite my initial shock that the thing won’t accept xhtml-mp markup, so although it is a mobile device you can’t treat it as such (at least not the way I’d like to with a custome xhtml-mp view of a site) I can see a way past that as it makes a pretty fair stab at rendering the full site. With the zooming feature – the double tapping is just genius – and wifi connectivity meaning the bandwidth isn’t much of an issue, it really is feasible to browse most sites in the full fat version without needing a slimmed down markup.

But, if the site hasn’t been sympathetically designed – and the one big irritant is lines of text that are too long to be able to read in one screenfull so you have to keep scrolling backwards and forwards – it’s easy to get less than satisfied. And if you’re on an iPhone with limited (and expensive) bandwidth optimised graphics etc. wuold be a big bonus. And having seen sites specifically designed for the iPhone e.g. iphone.facebook.com – they’re just so much nicer to use than the full one.

I’m working on a concept proving site at the moment from first principles, using tera-wurfl for device detection, and Smarty templates to deliver different styling of the same content to different devices. I’ve not yet settled on my preferred approach but when I do I’ll publish it somewhere.

One thing is for certain – I’m now more comfortable with the idea that you don’t treat iPhone (or iPod Touch) the same as a ‘normal’ phone, but don’t think you should ignore it either and feed it a site designed for big screens.

Future of Mobile

Filed under: mobile,review — jaydublu @ 6:44 pm

Future of Mobile, London IMAX Cinema, 14th November 2007

The fact that I didn’t have a laptop with me is partly why this post is a bit late, the other is how long it’s taken me to sort out my longhand scribbled notes. So the fact that I didn’t blog from my phone on the train on the way home may indicate the Future of Mobile is ‘not yet’

Hats off to Ryan and all at Carsonified for another excellent conference – I feel privileged to be able to attend these things in my home country rather than having to read about happenings in the States. It was a thought provoking and informative, if long day.

So what did I ‘take home’?

One Web – the future of te Internet is mobile. Adapt the experience to exploit device capabilities maintaining thematic consistency across all the various ‘screens’ (PC / mobile / TV etc.)

Standards are crucial – but some people are stubbornly doing their own thing.

iPhone will revolutionise the market – not just because of the UI / design, but because it’s setting new consumer expectations of mobile devices as more than a phone, and challenging the established mobile business model (carriers / operators control)

Convergence is an assumption – actual usage is multiple specific devices for specific tasks (e.g. phone, Blackberry, satnav each with a SIM) and different people want different things maintaining fragmentation.

Platforms are maturing – including the upcoming Open Handset Alliance. Diverse but maturing.

Exploit device potential – rich user experience not lowest common denominator – media query / WURFL / W3C core vocabulary. Mobile web experience is still poor ‘like kissing a girl who’s leaning away from you’ (Tony Fish)

Content may be King but context rules – location based services still unjustified hype, mobile search results are poor, massive security / privacy concerns.

But overriding all of that – due to the sheer scale of the market (more than the whole population of China) the potential for a successful application is massive.

As John Connor said: ‘I can’t help you with what you must soon face, except to say that the future is not set.

It’s all to play for!

PRTG Traffic Grapher

Filed under: review,tinkering — jaydublu @ 6:03 pm

It’s not often I come across something that solves a need so completely – but it happened yesterday when I renewed my acquaintance with PRTG Traffic Grapher.

The situation: We have two networks behind a firewall – our Internal ‘trusted’ network and a DMZ for dev servers. There can be around 50 people in the office daily – that adds up to probably 100+ devices all chatting away to each other and the Internet.

Our network has grown slowly over time as the Company has expanded, but we’ve tried to keep up with things and maintain stability and performance. There’s as solid a backbone as we can get without a major restructuring, with as little ‘daisy chaining’ of switches etc as we can arrange, and things are as logical and tidy as is feasible in a ‘real’ environment with things moving all the time.

Most of the time things are tickety-boo, but every now and again things grind to a sticky mess – like when I went in yesterday morning to cries of ‘where’s the Internet?’ and ‘Why can’t I get my email?’. After a process of elimination and some realitively crude diagnostics (unplugging things to see if there’s an improvement) it was tracked down to a single machine, that either had a dodgy network card, or the cabling from the patch panel to the machine was dodgy.

Next thing that happened was the Internet started slowing down, and looking at the bandwidth stats on the Firewall showed that we were pulling almost all of our 2mbps bandwidth constantly. No idea what it’s doing or who’s doing it, but it’s going into the ‘trusted’ network somewhere.

Every time this happens I resolve to find a monitoring solution to give more visibility of what’s happening on the network, so we have advance waring of problems, and the fault finding is more scientific.

We’ve tried various SNMP tools in the past but haven’t been able to get really useful results out of the mass of irrelevant data. We’re currently also using Nagios to monitor things, but again SNMP isn’t quite as useful as one might think to see where traffic is in a network. It also cannot distinguish between traffic inside the network (e.g. accessing network shares) and that leaving the network (e.g. downloading off the Internet)

Then I stumbled across PRTG, and if it did what the site said it did it sounded ideal, so fighting to get a bit of bandwidth (it took almost half an hour to download a 13MB file!) I installed it and started playing. First thing to set up was packet sniffing, which is great in that it breaks down all traffic on a monitored network interface into protocols and graphs it.

The problem is that in a switched network the only traffic getting to my machine (doing the monitoring) is that intended for my machine, or broadcast packets. Then I remember that our Netgear ‘Smart Switches’ have a monitoring function where all traffic to one port can be mirrored to another. So I set it up to send everything that goes to the uplink port to the one my machine is connected to – bingo! I can see all traffic leaving my switch.

What I then do is install the software on a server on the same switch as the link out of the network to the firewall, and that allows monitoring of all traffic into and out of the network. With a quick few filters to ignore connections between devices inside the network, or between trusted and optional, it leaves the stuff that in all likelihood is going to be leaving the network, and there’s a fantastic graph showing what’s going on, and giving information on what connections are using the most bandwidth. No pulling out patch cables until the bandwidth stops – this shows what it is and lets us decide if it’s legit traffic and there are just too many users sharing a single internet connection, or if someone is being greedy.

Yes it does SNMP too, which I’ve started configuring to monitor traffic between switches to check our backbone is OK, but I’ve got a very useful tool now for my biggest headache – ‘Who’s using all our bandwidth?’

SPGM – Simple Picture Gallery Manager

Filed under: review,web development — jaydublu @ 6:26 pm

My other half makes cakes, and like a good hubby I did her a website some time back – but like the old saying about cobblers children it should really be a bit more impressive than it is.

The main part is the gallery, and I started off with something I knocked up in an hour or so, but recently thought it needed some attention – didn’t look very good or work well, and was a nightmare to maintain.

So, the old quandary – do I start from scratch, search in vain for the perfect thing, or find something that’s close and adapt it (or put up with the failings)?

I went through a few gallery packages, and they either were to huge and clonky, or didn’t work (or I couldn’t be arsed to fiddle to make them work) or they had entirely the wrong feature set so I’d have to turn too much off. I didn’t want comments or other UGC / sharing / web 2.0 type features, I just wanted a nice simple way to display categorised groups of images with a little bit of description.

And since the images were being taken on a 6mp camera, and we didn’t want / need the full size image online, there should be an easy way to do the resizing / thumbnail generation etc.

And it should be light, and simple, and … nice.

I was just about to give up the search and write my own – I was thinking ftp images up into folders and have a very simple script that if it saw a new image would just display it, and if it found some meta-data in a text file or some such would disply that alongside. No need for databases or anything complex.

And then I found SPGM – and I found that someone had already written it. I’ve commented before on my love/hate with open source, but this is defintely on the love side. It’s just the thing, and almost exactly what I had in mind – in fact better because I didn’t have to write it.

So I install it, and it works first time, and I start thinking “OK, it’s great, but it could do with a basic tool to help with thumbnail creation and editing the meta-data” and then I read on and found that had been written too – SPGM Webministration – it was light and it worked first time and was just the thing. Two for two!

So I’ve knocked up a quick Firewoks macro to automate the process of generating 640×480 images to upload (don’t want to waste too much bandwidth by having the conversion done on the server) and 100 odd images are now online and looking funky(ish), in practically no time. And I start thinking “I wonder if there’s a tool to do that side of things too” and I check, and there is.

But I haven’t tried that one yet – no sense tempting fate.

Next Page »