Friday, April 27, 2012

"Why do you people always question? Why ask why, when *how* is so much more fun?"

    It's been awhile again since my last post, but as always there were many issues to deal with along the way. I would have to start with the technical changes (surprise, surprise). The diskless OpenBSD IPv6 router that I was blogging about in my previous post is obsolete now, replaced by a diskless FreeBSD version because as we saw when certain circumstances were present the OpenBSD would just crash (kernel coredump upon NFS failure for example) which is not tolerable so I rebooted it under FreeBSD (since I have a few OSes ready for netboot) and all the problems are gone. You can see some basic stats about our IPv6 connectivity and usage at http://IPv6.onlineDirect.bg. We also added new project to the mirrored ones called Sagemath and you can access it at http://Sagemath.Igor.onlineDirect.bg as usual. Continuing with the tradition of providing new services for our clients I decided to run our own Counter-Strike servers since most of the public ones are extremely crappy, I've added the minimum add-ons so the game can be enjoyable and also optimized the server a lot so the latency should be minimal. You can play CS Classic at cs.onlinedirect.bg:27015 and CS Italy2 Respawn at cs.onlinedirect.bg:27016.
    Another new development would be the new advanced DPI engine for my router which extremely accurately is able to differentiate between numerous protocols and if configured apply QoS based on some portion (or portions for that matter, it can be composed of few different parts) of the packet. We are using it to throttle "bad" traffic in favor of the "normal" user traffic so the service we provide would be optimal all the time. Currently I've implemented it as hybrid system doing the analysis in user-space and the matching/QoS in my router in kernel-space. I had to modify heavily the tree structures that we use because of the space requirements (there can be millions of concurrent matches per single machine) so they are as compact as possible. My tests show that currently for a little over 1 Mpps (that's our international traffic) about 4 - 6 % of the CPU is used for analysis (of course if a stream is analyzed it is cached etc) with superb accuracy and the improvement of our service is very noticeable. I've also added new user-space tools so one can track how the DPI engine works and see some statistics about it in real-time. Oh and the router now supports new load-balancing algorithm which can use up to 10 x 1/10 gbps network adapters in a single trunk.
   In the spirit of improvements I also decided to finally fix a few little issues with my BGPv4 implementation and in the process I also developed new peer balancing algorithm (which is very simplified version of the one I was writing paper about "Using combinatorics to load-balance Internet traffic") and results have jaw dropping effect. The Internet traffic got balanced evenly between all of our peers with difference margin of at worst 15 % (which means extremely well). I've added various triggers to the BGPv4 software which go off upon changes in the state of the peer and the balancing operation is fully automated. Another huge improvement in the BGPv4 software is the better use of tree structures to minimize CPU-time and now when a peer "dies" or is connected you can't see the BGP on the "top" process list for more than a second (even then taking 5 % of CPU time). I've added locking for the asynchronous events since I found a bug there which in very rare cases caused it to crash. All in all after all the changes I haven't had to log on to the machine that it's running on for any reason other than monitoring it.
    Oh wait! There's even more! :-) There's some news about the IOlib I mentioned in some of my previous posts. I've been actively developing it to add cross-platform compatibility and the results are very satisfying as the current state is:
OSes tested: *BSD (kqueue / poll), Linux (epoll / poll), Solaris (/dev/poll, poll)
IPv4 TCP/UDP sockets
IPv6 TCP/UDP sockets
Unix domain sockets
FILE I/O
SSL sockets (using OpenSSL)
I've easily managed to make a daemon that sustained over 200 000 active connections transferring over 500 MB/s with very little effort (most of which went into optimizing kernel tunables). When I add all the protocols and functionality that I wish for such a library I will release it as open source (with no license - simply free).
    On a personal note I got myself an SSD disk (OCZ Vertex 3) and I'm in heaven right now! This is by far the best upgrade one can make. As my CPU stayed idle because of my old slow HDD, now I can really feel the speed (I upgraded my PC to Sandy Bridge Core i5-2500 with 1600 MHz DDR3 memory). Boot time is little over 10 seconds without optimizing the operating system start-up (only removed few things from the bootloader), everything just pops up immediately. Since my work is connected with a lot of kernel recompilation the Linux kernel with default config from Debian compiled for about 10 minutes, bonnie++ showed 700 MB/s read speed and  nearly 500 MB/s write speed when tested. The productivity went through the roof with this upgrade and working on the PC is a lot more pleasant :-)
   But wait! There's more, I moved all my sources in a git repository that I setup on my storage. I'm pleasantly surprised by how well it works and how easy it is to develop software by using it. I've configured different SSH keys for various access types and also added active backup. In this spirit (it is a Bulgarian saying) I decided to switch to a more centralized identification, authorization and accounting system using OpenLDAP, Kerberos and the ldap pam module. I also played around with the Single Sign On idea using Kerberos, I even locked myself out of the machines one time so it's been fun :-) but I prefer more paranoid approaches to security.
   Some good news about the ISP is that we won the project we had applied for and now let there be PON service :-) In my next post I'll divulge more about this one, there would be some really big and nice new toys to play with. I'm hoping for 10G NetFPGA dev boards!
   It was about time I generate myself a PGP keypair :-)
My key id is: E01113B1
Keyserver: pgp.mit.edu
I also updated my CV and started using Dropbox (very convenient by the way) so you can find my CV in PDF format at http://dl.dropbox.com/u/75462401/CV.pdf

That's it for now! (since there're always a lot of ideas before writing and while writing they all disappear)
 
P.S. One interesting project that I'm working on is an ARC cache implementation for a VM subsystem :-)

P^2.S. The title is a quote from the cult movie "Spawn"

P^3.S. You can access my FreeBSD's loader.conf & sysctl.conf these are used on one of the most loaded servers in terms of jails, services & traffic. The machine is performing great (it has network and file i/o load). You can see info about the machine at http://Igor.onlineDirect.bg.