-
Announcing Percona Toolkit Release 2.0.3
We’ve released Percona Toolkit 2.0.3, with a couple of major improvements and many minor ones. You can download it, read the documentation, and get support for it.
What’s new? You can read the changelog for the details, but here are the highlights:
Brand new pt-diskstats, thanks to Brian Fraser. This tool is completely rewritten, and it’s finally the iostat replacement I always wanted. Not only does it have the functionality I want (interactive, slice and dice, smart defaults) but it has the detailed statistics on I/O, so you can see whether your reads are slow versus your writes, and whether things are waiting on the disk or waiting on the queue scheduler (cfq, hint hint). Finally, it has transparency, so you can read the documentation and understand, really, what it’s doing at the low level and what that means for your server. We really need specific, precise information on exactly how the I/O is behaving so we can make good decisions when there are problems or when doing things like capacity planning.
Brand new pt-stalk, courtesy of Daniel Nichter. This tool is also completely rewritten. Instead of a Bash script that you have to configure with environment variables and run in a screen session, this is now a first-class fault detection daemon. Everyone needs post-mortem forensic data when there is a problem, and pt-stalk aims to be a core part of your infrastructure that fills this gap. It now supports things a “real” Percona Toolkit tool ought to have, such as command-line options and a configuration file. In addition, we merged pt-collect into it, so as of Percona Toolkit 2.0.3, there isn’t a separate pt-collect tool anymore.
There’s a lot more to this release, but those are the major points. Download it and let us know how it works, please! If you find bugs, file them on Launchpad, and if you need support, you know where to get it.
-
SpyMemcached Transcoder with PHP PDO Objects using ZLIB
My technology stack services more then 2 Million Daily Active users. Its very basic. PHP talks to mySQL, Memcache, RabbitMQ, Gearman and Facebook. Now that we have more Java specifically to support our SmartFox Server and other services, blurring the lines between what data is set in PHP and what data is read in Java is very necessary.Java J-Connect makes reading mySQL Data as simple IMHO as PHP's PDO. What is hard is reading PHP's serialized format from PHP's Memcache library.In PHP there are two main C backed Libraries. There is Memcache the original PHP library which I happen to use, and Memcached which is the library I wanted to use but didn't deploy since EC2 package system conflicted and cause issues (I fixed them but to late to deploy). Memcache stores data in PHP's serialized format and compresses it via ZLIB, while Memcached can store data as PHP's serialized format, JSON, Binary Serialized (which is rather awesome), JSON Array Notation and has a multitude of compressing formats none of which are pure ZLIB that I noticed.Here is the problem. Spymemcached is a lib for talking to memcache but can't unserialized PHP serialized format (or read it natively and return a string) and cannot Decompress ZLIB but can Decompress GZIP. Now a great speed up would be to use PHP's serialized data set from PHP and share memcache resources from PHP and Java just like what is done for the mySQL resources.What needs to be done? Well, build your own Transcoder for Spymemcached. Fortunately Spymemcached documented an interface to do just that.What is needed. Implement the spymemcached Interface defined here. Use org.lorecraft.phparser to unserialize PHP data defined here. Return the Object. Below is the code.package com.schoolfeed.spymemcached;import java.io.ByteArrayInputStream;import java.io.ByteArrayOutputStream;import java.io.IOException;import java.util.zip.InflaterInputStream;import net.spy.memcached.CachedData;import net.spy.memcached.compat.CloseUtil;import net.spy.memcached.transcoders.BaseSerializingTranscoder;import net.spy.memcached.transcoders.Transcoder;import org.lorecraft.phparser.*;public class PHPSerializedTranscoder extends BaseSerializingTranscoder implements Transcoder <Object> { static final int COMPRESSED=2; /** * Get a serializing transcoder with the default max data size. */ public PHPSerializedTranscoder() { this(CachedData.MAX_SIZE); } /** * Get a serializing transcoder that specifies the max data size. */ public PHPSerializedTranscoder(int max) { super(max); } /** * decode the byte data from Memcache decompress it if necessary and return the Object * @param CacheData - the byte data is turned into a object * @returns Object */ public Object decode(CachedData d){ byte[] data=d.getData(); Object rv=null; String ds="N;"; if((d.getFlags() & COMPRESSED) != 0) { getLogger().debug("Looks like d is compressed"); data=decompress(d.getData()); } ds=decodeString(data); getLogger().debug("DECODED: [" + ds + "] about to SerializedPhpParser"); SerializedPhpParser sp = new SerializedPhpParser(ds); try { rv = sp.parse(); getLogger().debug("Parse was cool!!"); } catch(Exception e){ getLogger().debug("Not a PHP Object? : " + ds); rv = ds; } return rv; } /** * PHP Memcache stores compress data in ZLIB format override the base class decompress method to handle ZLIB * * @param byte array - raw data from Memcache * @returns byte array */ protected byte[] decompress(byte[] in) { ByteArrayOutputStream bos=null; final int BUFFER = 2048; if(in != null) { ByteArrayInputStream bis=new ByteArrayInputStream(in); bos=new ByteArrayOutputStream(); InflaterInputStream iis = null; try { iis = new InflaterInputStream(bis); byte[] buf=new byte[BUFFER]; int r=-1; while((r=iis.read(buf, 0, BUFFER)) > 0) { bos.write(buf, 0, r); } } catch (IOException e) { getLogger().warn("Failed to decompress data", e); bos = null; } finally { CloseUtil.close(iis); CloseUtil.close(bis); CloseUtil.close(bos); } } return bos == null ? null : bos.toByteArray(); } /** * encode -- not implemented yet * */ public CachedData encode(Object o){ int flags = 0; byte[] b=null; return new CachedData(flags, b, getMaxSize()); } /** * no need to async Decode let's do it realtime */ public boolean asyncDecode(CachedData d) { return false; }}This is a stop-gap solution until we make the transition to Memcached with JSON encoding. Then I can use Jackson-JSON - which is a fast JSON encoder/decoder for Java enabling a great portable message protocol between the two stacks and nearly any other language we might add to the system (like Python).
-
Dot-Org Pavilion at the Percona Live MySQL Conference
Are you involved with an open-source project that’s interesting to MySQL users, such as Nginx, PHPMyAdmin, Drupal, Jenkins, PHP, and so on? Percona just published the application form for dot-org groups to have a free expo hall booth in the Percona Live MySQL Conference in April. Please submit your applications now, and tell your friends about this, because a) the schedule for applying is very short, and b) space is limited.
For those of you who don’t know what this is, it’s another of the O’Reilly traditions we’re trying to continue. (We are trying very hard to make this event as close to a clone of O’Reilly’s as we can.) It’s a free table in the expo hall where people who participate in a non-commercial open source project can exhibit. I organized a Maatkit booth a few times in the past, and was always really grateful to O’Reilly for making the space available. Space in the expo hall is at a premium, but we think that these dot-org booths are even more valuable to the open-source projects and the conference attendees.
So, please tell your friends who care about open source, and ask them to tell their friends too. Let’s get some great open-source projects into the expo hall, alongside the commercial vendors!
Further Reading:Percona Live MySQL Conference comes to Washington DC
One-day Percona Live conference in San Francisco
Maatkit at the dot-org pavilion at MySQL Conference and Expo 2009
Schedule for MySQL-and-beyond conference is live
Percona Live from a community insider’s perspective
-
MySQL progress in a year
Usually people do this around New Year, I will do it in February. Actually, I was inspired to do this after reviewing all the talks for this year's MySQL Conference - what a snapshot into the state of where we are! It made me realize we've made important progress in the past year, worth taking a moment to celebrate it. So here we go...
Diversification
In the past few years there was a lot of fear and doubt about MySQL due to Oracle taking over the ownership. But if you ask me, I was more worried for MySQL because of MySQL itself. I've often said that if MySQL had been a healthy open source project - like the other 3 components in the LAMP stack - then most of the NoSQL technologies we've seen come about would never have been started as their own projects, because it would have been more natural to build those needs on top of MySQL. You could have had a key-value store (HandlerSocket, Memcache API...), sharding (Spider) and clustering (Galera). You could have had a graph storage engine (OQGraph engine isn't quite it, I understand it is internally an InnoDB table still?). There could even have been MapReduce functionality, although I do think the Hadoop ecosystem targets problems that actually are better solved without MySQL.
read more
-
Upcoming talks at PHP UK, Confoo and your event
I will be speaking on new MySQL features at the PHP UK conference on February 25th in London. A few days later on march 3rd, I will be presenting the Care and Feeding of a MySQL Database at Confoo in Montreal. If you are attending either event, please introduce yourself!
Fellow Community Manager Keith Larson and I working hard to keep up with all the upcoming shows. If you think we might have not notices your upcoming event, please drop us a line. We will have thumb drive, baggage tags, stickers, and maybe (fingers crossed) t-shirts or plush dolphins soon.
|