Welcome to 2015!
I started writing code in 1976 when the z80 microprocessor came out, back then it was assembly language and tools were primitive. In 1982 I learned “C” and in the 1990’s I learned C++, C#, Java and PHP, by then the languages and OO programming had exploded. I still do a lot ofPHP and Shell Scripting and the occasional C++ app so this blog has some bits and pieces I often find useful and I will be ramping up the article index as I find time to write about stuff I’m working on.
I find I also do a lot of Javascript coding using JQuery and Ive been playing with D3.js as part of my work for offgrid Engineering and the development of the stock-punter.com web site which will use it heavily.
Recently I removed all the older Javascript pages from this BLOG, they are obsolete and I dont encourage anyone to use them. A new article on writing modern JS code will feature soon 🙂
Right now I’m learning D3.js and have a few projects in mind for it. I’m also using Laravel the PHP Framework and its brilliant, all the best components from other frameworks bundled and integrated. I’m so impressed with it I have started writing articles on getting up and running with Laravel.
Large Scale Engineering Projects
In 2000 I engineered an IBM Websphere Message Queue based system for the distribution of encrypted medical data for a large scale OLTP laboratory system, I’m proud to say after 15 years its still running non-stop and has sent BILLIONS of messages a year and has shown to scale as the business grew, added new labs and 1000’s of new clients. According to the IBM Websphere Product rep prior to me leaving that organization, the system was/is still the largest medical data messaging system in the southern hemisphere if not in the world when I stopped working on it in 2007.
I recently caught up with the support team who still look after it over a coffee in September 2014 and its still running not stop and has over 18,000 queues! – Boosting my philosophy that the world works on queues.
While on queue’s, I’ve previously reviewed a book called..”Rabbit MQ Cookbook” from Packt Publishing, here is a link to it and other RabbitMQ titles. So far it reads well and there is a good section on using Amazon Web Services, I need to make notes and send them back to the publisher but more on the book later.
These days I work as a Senior Systems Engineer/Systems Administrator for a boutique Cloud Hosting company who has a growing container based infrastructure on top of Cloud storage (where 3 copies of everything are kept and there is no RAID disks). I’m in the process of expanding a cloud platform (based on Cloud Linux) that I built when I started there in 2013. It uses a very neat distributed clustered storage implementation. See my post on PCS Storage.
Monitoring using Queues
One of the challenges of the cloud platform I have put together has been to monitor the hardware nodes, the Linux containers running on them and the individual applications running in each container. I’ve chosen RabbitMQ to be the backbone with a Collectd style messaging concept but driven from the client not the server. Initially I tried to implement my old favourite Nagios but it didn’t scale well in a hosting environment.
This should allow me to scale the hardware events, container events and application events into different servers and then have other processes collect and process the messages as they come through the queues.
As far as I see it removes any latency from the Server driven model of Nagios/Icinga etc and makes the data collection real time as possible while allowing the server to spend its time processing the data not gathering it and waiting for the client.
Magento MQ – Look out for it in 2015 2016
I was going to work on this in 2015 but I’ve been so busy on Laravel platforms I have had no time!
One of the issues I find with Magento is its performance. It can be incredibly slow because it has to process so much data in XML and layout files each time the user moves around the web interface. At checkout time, the processing cannot be cached so the engineering of Magento has moved me to design a Rabbit MQ solution to dissect and move the Async operations away from the Magento core. Effectively they can be offloaded from the main web server and it should be easier to implement in a High Availability environment using HAProxy or a clustered filesystem and distributed apps or some other similar architecture.
Re-engineering Magento to use RabbitMQ should add considerable performance gains and move it towards even better HA/Distributed scalable models. So far its a design concept with no code, I’m coding other RabbitMQ based apps at present in “C++” and when I’m finished that project I’ll start on the Magento port.
AMQP Libraries
Another topic I’m working on right now is coding using some of the available RabbitMQ libraries. I’ll post some article on these with working examples as I go. so far I have a major private project on the go where everything is queue based and the clients use the various AMQP public libraries to collect and process data from the queues… more to follow. 🙂
So the project scope for 2015 is:
- RabbitMQ based Monitoring system
- Magento MQ
- D3 graphing for stock-punter.com
- Docker infrastructure for container-hosting.com
z900collector 