Software/Sys Engineering


Welcome to 2014!

I started writing code in 1976 when the z80 microprocessor came out, back then it was assembly language and tools were primitive. In 1982 I learned “C” and in the 1990’s I learned C++, C#, Java and PHP, by then the languages and OO programming had exploded. I still do a lot of C++, PHP and Shell Scripting so this blog has some bits and pieces I often find useful and I will be ramping up the article index as I find time to write about stuff I’m working on.

Large Scale Engineering Projects

In 2000 I engineered a IBM Websphere Message Queue based system for the distribution of encrypted medical data for a large scale OLTP laboratory system, I’m proud to say after 14 years its still running non stop and send BILLIONS of message a year and has shown to scale as the business grew, added new labs and 1000’s of new clients. According to the IBM Websphere Product rep prior to me leaving that organization, the system was/is still the largest medical data messaging system in the southern hemisphere if not in the world when I stopped working on it in 2007.

I recently caught up with the support team who still look after it over coffee a few months ago and its still running not stop and has over 18,000 queues! – Boosting my philosophy that the world works on queues.

While on queue’s, I’m currently reviewing a new book called..”Rabbit MQ Cookbook” from Packt Publishing, here is a link to it and other RabbitMQ titles. So far it reads well and there is a good section on using Amazon Web Services, I need to make notes and send them back to the publisher but more on the book later.

These days I work as a Senior Systems Engineer/Systems Administrator for a boutique ISP who has a growing cloud container based infrastructure. I’m in the process of deploying a new cloud platform (based on Cloud Linux) using a very neat distributed clustered storage implementation, with that will come a container migration project and then a big drive into the Magento hosting and support space.

Monitoring using Queues

One of the challenges of the cloud platform I have put together has been to monitor the hardware nodes, the Linux containers running on them and the individual applications running in each container. I’ve chosen RabbitMQ to be the backbone with an Nagios/NRPEĀ  style messaging concept but driven from the client not the server. This should allow me to scale the hardware events, container events and application events into different servers and then have other processes collect and process the messages as they come through the queues.

As far as I see it removes any latency from the Server driven model of Nagios/Icinga etc and makes the data collection real time as possible while allowing the server to spend its time processing the data not gathering it and waiting for the client.

Magento MQ – Look out for it in 2014

One of the issues I find with Magento is its performance. It can be incredibly slow because it has to process so much data in XML and layout files each time the user moves around the web interface. At checkout time, the processing cannot be cached so the engineering of Magento has moved me to design a Rabbit MQ solution to dissect and move the Async operations away from the Magento core. Effectively they can be offloaded from the main web server and it should be easier to implement in a High Availability environment using HAProxy or a clustered filesystem and distributed apps or some other similar architecture.

Re-engineering Magento to use RabbitMQ should add considerable performance gains and move it towards even better HA/Distributed scalable models. So far its a design concept with no code, I’m coding other RabbitMQ based apps at present in “C++” and when I’m finished that project I’ll start on the Magento port.

AMQP Libraries

Another topic I’m working on right now is coding using some of the available RabbitMQ libraries. I’ll post some article on these with working examples as I go. so far I have a major private project on the go where everything is queue based and the clients use the various AMQP public libraries to collect and process data from the queues… more to follow. :)

So the project scope for 2014 is:

  • RabbitMQ based Monitoring system
  • Magento MQ

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s