Welcome to 2014!
I started writing code in 1976 when the z80 microprocessor came out, back then it was assembly language and tools were primitive. In 1982 I learned “C” and in the 1990′s I learned C++, C#, Java and PHP, by then the languages and OO programming had exploded. I still do a lot of C++, PHP and Shell Scripting so this blog has some bits and pieces I often find useful and I will be ramping up the article index as I find time to write about stuff I’m working on.
Large Scale Engineering Projects
In 2000 I engineered a IBM Websphere Message Queue based system for the distribution of encrypted medical data for a large scale OLTP laboratory system, I’m proud to say after 14 years its still running non stop and send BILLIONS of message a year and has shown to scale as the business grew, added new labs and 1000′s of new clients. According to the IBM Websphere Product rep prior to me leaving that organization, the system was/is still the largest medical data messaging system in the southern hemisphere if not in the world when I stopped working on it in 2007.
I recently caught up with the support team who still look after it over coffee a few months ago and its still running not stop and has over 18,000 queues! – Boosting my philosophy that the world works on queues.
While on queue’s, I’m currently reviewing a new book called..”Rabbit MQ Cookbook” from Packt Publishing, here is a link to it and other RabbitMQ titles. So far it reads well and there is a good section on using Amazon Web Services, I need to make notes and send them back to the publisher but more on the book later.
These days I work as a Senior Systems Engineer/Systems Administrator for a boutique ISP who has a growing cloud container based infrastructure. I’m in the process of deploying a new cloud platform (based on Cloud Linux) using a very neat distributed clustered storage implementation, with that will come a container migration project and then a big drive into the Magento hosting and support space.
Monitoring using Queues
One of the challenges of the cloud platform I have put together has been to monitor the hardware nodes, the Linux containers running on them and the individual applications running in each container. I’ve chosen RabbitMQ to be the backbone with an Nagios/NRPE style messaging concept but driven from the client not the server. This should allow me to scale the hardware events, container events and application events into different servers and then have other processes collect and process the messages as they come through the queues.
As far as I see it removes any latency from the Server driven model of Nagios/Icinga etc and makes the data collection real time as possible while allowing the server to spend its time processing the data not gathering it and waiting for the client.
Magento MQ – Look out for it in 2014
One of the issues I find with Magento is its performance. It can be incredibly slow because it has to process so much data in XML and layout files each time the user moves around the web interface. At checkout time, the processing cannot be cached so the engineering of Magento has moved me to design a Rabbit MQ solution to dissect and move the Async operations away from the Magento core. Effectively they can be offloaded from the main web server and it should be easier to implement in a High Availability environment using HAProxy or a clustered filesystem and distributed apps or some other similar architecture.
Re-engineering Magento to use RabbitMQ should add considerable performance gains and move it towards even better HA/Distributed scalable models. So far its a design concept with no code, I’m coding other RabbitMQ based apps at present in “C++” and when I’m finished that project I’ll start on the Magento port.
Another topic I’m working on right now is coding using some of the available RabbitMQ libraries. I’ll post some article on these with working examples as I go. so far I have a major private project on the go where everything is queue based and the clients use the various AMQP public libraries to collect and process data from the queues… more to follow. :)
So the project scope for 2014 is:
- RabbitMQ based Monitoring system
- Magento MQ