Welcome to 2015!
Right now I’m learning D3.js and have a few projects in mind for it.
Large Scale Engineering Projects
In 2000 I engineered a IBM Websphere Message Queue based system for the distribution of encrypted medical data for a large scale OLTP laboratory system, I’m proud to say after 15 years its still running non-stop and has sent BILLIONS of message a year and has shown to scale as the business grew, added new labs and 1000’s of new clients. According to the IBM Websphere Product rep prior to me leaving that organization, the system was/is still the largest medical data messaging system in the southern hemisphere if not in the world when I stopped working on it in 2007.
I recently caught up with the support team who still look after it over coffee in September 2014 and its still running not stop and has over 18,000 queues! – Boosting my philosophy that the world works on queues.
While on queue’s, I’m currently reviewing a new book called..”Rabbit MQ Cookbook” from Packt Publishing, here is a link to it and other RabbitMQ titles. So far it reads well and there is a good section on using Amazon Web Services, I need to make notes and send them back to the publisher but more on the book later.
These days I work as a Senior Systems Engineer/Systems Administrator for a boutique ISP who has a growing cloud container based infrastructure. I’m in the process of expanding a cloud platform (based on Cloud Linux) that I built when I started there in 2013. It uses a very neat distributed clustered storage implementation. See my post on PCS Storage.
Monitoring using Queues
One of the challenges of the cloud platform I have put together has been to monitor the hardware nodes, the Linux containers running on them and the individual applications running in each container. I’ve chosen RabbitMQ to be the backbone with an Nagios/NRPE style messaging concept but driven from the client not the server. This should allow me to scale the hardware events, container events and application events into different servers and then have other processes collect and process the messages as they come through the queues.
As far as I see it removes any latency from the Server driven model of Nagios/Icinga etc and makes the data collection real time as possible while allowing the server to spend its time processing the data not gathering it and waiting for the client.
Magento MQ – Look out for it in 2015
One of the issues I find with Magento is its performance. It can be incredibly slow because it has to process so much data in XML and layout files each time the user moves around the web interface. At checkout time, the processing cannot be cached so the engineering of Magento has moved me to design a Rabbit MQ solution to dissect and move the Async operations away from the Magento core. Effectively they can be offloaded from the main web server and it should be easier to implement in a High Availability environment using HAProxy or a clustered filesystem and distributed apps or some other similar architecture.
Re-engineering Magento to use RabbitMQ should add considerable performance gains and move it towards even better HA/Distributed scalable models. So far its a design concept with no code, I’m coding other RabbitMQ based apps at present in “C++” and when I’m finished that project I’ll start on the Magento port.
Another topic I’m working on right now is coding using some of the available RabbitMQ libraries. I’ll post some article on these with working examples as I go. so far I have a major private project on the go where everything is queue based and the clients use the various AMQP public libraries to collect and process data from the queues… more to follow. :)
So the project scope for 2015 is:
- RabbitMQ based Monitoring system
- Magento MQ
- D3 graphing for stock-punter.com
- Docker infrastructure for container-hosting.com