tag:blogger.com,1999:blog-139658742024-03-07T15:21:47.959-08:00The WallBrendan Gregg's personal blog.Brendan Gregghttp://www.blogger.com/profile/03662900777558310512noreply@blogger.comBlogger44125tag:blogger.com,1999:blog-13965874.post-83430597811561697712015-05-15T13:32:00.000-07:002015-05-16T15:55:21.297-07:00The DTraceToolkit Project Has Ended<div dir="ltr" style="text-align: left;" trbidi="on">
<p>Ten years ago on this day I created the DTraceToolkit, and it's time to call the project ended. Its scripts live on in different operating systems: OS X, FreeBSD, Oracle Solaris 11, and other Solaris derivatives; as packages for OmniOS and SmartOS; and integrated into other tools. Thanks to everyone who helped make it a success.</p>
<p>I documented its origin in the History file:</p>
<pre>$ more DTraceToolkit-0.99/Docs/History
------------------------------------------------------------------------------
20-Apr-2005 Brendan Gregg Idea
For a while I had thought that a DTrace toolkit would be a nice
idea, but on this day it became clear. I was explaining DTrace to
an SSE from Sun (Canberra, Australia), who had a need for using
DTrace but didn't have the time to sit down and write all the
tools he was after. It simply made sense to have a DTrace toolkit
that people could download or carry around a copy to use. Some
people would write DTrace tools, others would use the toolkit.
------------------------------------------------------------------------------
15-May-2005 Brendan Gregg Version 0.30
I had discussed the idea of a DTrace toolkit with the Sun PAE guys in
Adelaide, Australia. It was making more sense now. It would be much
like the SE Toolkit, not just due to the large number of sample
scripts provided, but also due to the role it would play: few people
wrote SE Toolkit programs, more people used it as a toolkit. While
we would like a majority of Solaris users to write DTrace scripts,
the reality is that many would want to use a prewritten toolkit.
Today I created the toolkit as version 0.30, with 11 main directories,
a dozen scripts, man pages and a structure for documentation.
...
</pre>
<p>Back in 2005, the DTraceToolkit was a collection of robust performance tools for a single OS and kernel, providing advanced performance insight beyond the typical Unix toolset. I've had countless emails from sysadmins and developers who have used it to solve performance issues in production, and wanted to say thanks. I've appreciated the kind words!</p>
<p>Today, in 2015, the 230-script DTraceToolkit is more like a large collection of ancient kernel patches that, when they do work, often do so by sheer luck. I didn't know in 2005 that DTrace would appear on other operating systems, and that each kernel would change as much as it did. In particular there were numerous changes to the DTrace <a href="http://dtrace.org/blogs/brendan/2011/11/09/solaris-11-dtrace-syscall-provider-changes/">syscall provider</a>, which caused the scripts to be more tied to kernel versions than expected.</p>
<p>To put this all in today's terms: the DTraceToolkit became like a set of 230 <i>amazing</i> Linux 2.6.11 patches that people want working on Linux 3.2, 3.13, and 4.0, <i>and</i> FreeBSD 10.0, 11.0, <i>and</i> Oracle Solaris 11! Such a feat isn't impossible, in the strictest sense of the word, but it is impractical.</p>
<p>In 2013 I saw how to fix this, while keeping the DTraceToolkit as a central collection of scripts. The trick was realizing that there were two audiences with different requirements, who could be served by having two different collections of tools:</p>
<ul>
<li><b>A toolkit</b> of working and maintained tools for everyone to use. It should be easy to learn, providing simple Unix-like tools with man pages and example files. It should also provide the fewest tools possible (fewer than twenty), to make it easier to learn, browse, and maintain.</li>
<li><b>A toolshed</b> of in-development or unmaintained tools for performance engineers to browse. This would be a large library of hundreds of scripts, most of which won't work on any given kernel. These serve as ideas, suggestions, and starting points for performance analysis, and can be fixed when needed. (In a way, the DTrace book serves as this; its scripts are on <a href="http://dtracebook.com">dtracebook.com</a> and <a href="https://github.com/brendangregg/DTrace-book-scripts">github</a>.)</li>
</ul>
<p>I planned to do this split, and started explaining it in my 2013 post on <a href="http://bdgregg.blogspot.com/2013/09/dtracetoolkit-0xx-mistakes.html">DTraceToolkit 0.xx Mistakes</a>. It would be a lot of work. I was primarily interested in helping the toolkit users, but I had another minor motivation: to give a DTraceToolkit talk at the USENIX/LISA conference – a dream I'd had for years. Despite giving many talks, and including the DTraceToolkit in some of them, I'd never given a canonical DTraceToolkit talk.</p>
<p>However, it was not to be. In March 2014 I stopped working on Solaris or any of its derivatives, and accepted a job to work primarily on Linux performance. While the DTraceToolkit was always a spare time project, I have to admit that it's no longer a priority, and hasn't been for years. There are new and exciting things in tech to work on (including Linux eBPF), and which are more related to my day job and career going forward.</p>
<p>I still use some DTrace ... on FreeBSD, which is also in use at my new job. And last year I released a new set of (toolshed-like) tools for FreeBSD: the <a href="https://github.com/brendangregg/DTrace-tools">DTrace-tools</a> collection.</p>
<p>I expected my new job to be the most challenging of my career, and it has been. Early on I desperately missed the DTraceToolkit while on Linux, as well as my other DTrace tools. But I've been writing new ones, out of necessity, based on Linux ftrace, perf_events, SystemTap, and eBPF. I'm making progress, script by script, bringing the observability I need to Linux, and sharing these scripts online.</p>
<p>I gave my LISA talk after all, in 2014, but it was titled <a href="http://www.brendangregg.com/blog/2015-03-17/linux-performance-analysis-perf-tools.html">Linux Performance Analysis: New Tools and Old Secrets</a>. This was about my Linux ftrace and perf_events-based tools: <a href="https://github.com/brendangregg/perf-tools">perf-tools</a>, which are inspired by my own DTraceToolkit. It includes multi-tools like <a href="https://github.com/brendangregg/perf-tools/blob/master/examples/funccount_example.txt">funcccount</a> and <a href="https://github.com/brendangregg/perf-tools/blob/master/examples/kprobe_example.txt">kprobe</a>, which, for me, make a giant difference. I can rapidly explore kernel behavior again.</p>
<p>A number of the DTraceToolkit scripts live on in different OSes, and I'm glad for that to happen: they have a life of their own. But I can't recommend anyone continue the DTraceToolkit project. If you want to understand more background as to why, then my <a href="http://bdgregg.blogspot.com/2013/09/dtracetoolkit-0xx-mistakes.html">mistakes</a> blog post should be a good start.</p>
<p>I'm proud of what I accomplished with the DTraceToolkit, the DTrace book, and with Sun Solaris in the field of performance. Thank you Sun – you were awesome back in the day.</p>
<p>I'll be moving the DTraceToolkit into my <a href="http://www.brendangregg.com/crypt.html">Crypt</a>, which has my other retired Solaris software. It's not easy to do this, but I think it's better to communicate bad news than none at all.</p>
<pre>
dtrace -n 'END { printf("The %stoolkit has %sED\n", probeprov, probename); }'
</pre>
<p>Probably the best outcome of my DTraceToolkit work was my <a href="http://dtracebook.com">DTrace book</a> with Jim Mauro. I really think of it as the DTraceToolkit version 2.0.</p>
<p><i>PS. This is my last post to this blog, which mostly existed for DTraceToolkit updates. My new blog is <a href="http://www.brendangregg.com/blog/">here</a>.</i></p>
</div>
Brendan Gregghttp://www.blogger.com/profile/03662900777558310512noreply@blogger.com0tag:blogger.com,1999:blog-13965874.post-86279847465281191462013-09-05T21:56:00.000-07:002015-05-02T15:16:54.686-07:00DTraceToolkit 0.XX Mistakes<div dir="ltr" style="text-align: left;" trbidi="on">
<p>You learn more from failure than you do success. In this post, I'd like to list <b>my mistakes and failures</b> from versions 0.01 to 0.99 of the <a href="http://www.brendangregg.com/dtrace.html#DTraceToolkit">DTraceToolkit</a>, as lessons to learn from. There is much detail here that may only interest a small number of people. And if you are a recruiter, I can assure you that this won't interest you at all, so please stop reading. In fact, I'm <i>another</i> Brendan Gregg – the one who makes many mistakes – since there are <a href="http://www.brendangregg.com/Images/brendan_clones2006.jpg">several of us</a>, after all. :-)</p>
<p>As a summary of the lessons learned, skip to the end for the "Learning From Mistakes" section.</p>
<h3>Background</h3>
<p>By 2005 I was sharing a small collection of DTrace scripts – tools – on my <a href="http://www.brendangregg.com/dtrace.html">homepage</a>. These were popular with a wide audience, including those who didn't have time to write DTrace scripts from scratch, and those who weren't programmers anyway. For these casual DTrace users I created a toolkit, to:</p>
<ul>
<li>Give people useful tools to run immediately</li>
<li>Present good examples of DTrace to learn from</li>
</ul>
<p>I was doing performance consulting and training, and had a keen sense of what problems needed solving. So creating useful tools was easy: I already had a laundry list of needs. As for good examples: I made my own coding style and stuck to it. I also created documentation for every script: a man page, and a file showing example usage. And I tested every script carefully.</p>
<p>The toolkit has been successful, helping people solve issues and learn DTrace. The scripts have also have been included, as tools, in multiple operating systems by default, including Mac OS X and Oracle Solaris.</p>
<h3>Mistake 1. Missing Scripts</h3>
<p>The observability coverage was a little uneven, as it was based on my performance experience at the time. Some areas I didn't write enough scripts for, and some areas I missed by mistake: for example, socket I/O from the syscall interface. I should have drawn a functional diagram of the system and kernel internals, to look for areas that were lacking scripts and observability.</p>
<p>I did cover the missing areas when I wrote numerous new scripts for the <a href="http://www.dtracebook.com">DTrace book</a>, which are shared <a href="http://www.dtracebook.com">online</a> as its own collection. But I haven't figured out how to include them in the DTraceToolkit, as they were created as example scripts (that likely need tweaking to run on different OSes) and not robust tools. So, now I have two collections of scripts, and over 400 scripts total.</p>
<h3>Mistake 2. Too Many Scripts</h3>
<p>I think I have too many scripts for the target audience: casual users. I'm reminded of the function/complexity ratio described in The Mythical Man Month, where the addition of some function came at the cost of much more complexity, making computers harder to learn and use. Casual users may only use a handful of the scripts, and having extra scripts around adds complexity to browse. This also becomes a maintenance burden, and testing can fall behind (it already has).</li>
<p>I did use a directory hierarchy to organize the scripts, with "top scripts" in the top level directory, which I think helps. I've also been thinking of creating a separate collection for performance engineers which contains every script (400+), and reducing the DTraceToolkit to 100 or less for the casual users.</p>
<h3>Mistake 3. Inventing My Own Style</h3>
<p>I should not have invented my own style to begin with. This included double asterisks for block comments, and no spaces after commas. From the original <a href="http://www.brendangregg.com/DTrace/oldiosnoop.d">iosnoop</a>:</p>
<pre>
#!/usr/sbin/dtrace -s
/*
** iosnoop.d - A program to print I/O events as they happen, with useful
** details such as UID, PID, inode, command, etc.
** Written in DTrace (Solaris 10 build 51).
**
** 29-Mar-2004, ver 0.60. (check for newer versions)
[...]
</pre>
<p>At the time I thought it was neat. I now think it looks bad. After about 50 scripts, someone from Sun suggested I follow "cstyle", as that was the standard for Sun's C code. This seemed to be a better idea than my own invented style, but, I had already written 50 scripts! I had to rewrite all the scripts to be cstyled - a nusiance. I should have asked others in the DTrace community before creating my own style, as many would have made the same suggestion: use cstyle.</p>
<h3>Mistake 4. Complex scripts</h3>
<p>Some of the scripts are too long and too complicated. These make poor examples to learn DTrace from, which was one of the goals of the toolkit. They also are a pain to maintain. In some cases it was necessary, since the point of the script was to resemble a typical Unix *stat tool, and therefore needed shell wrapping (getopts). Eg, iosnoop, iotop, and execsnoop. But in other cases it wasn't necessary, like with tcpsnoop.d.</p>
<p>One reason tcpsnoop.d is complex is the provider it uses (next mistake), but another reason was the aim. I wanted a tool that emitted the exact same output as snoop (Solaris tcpdump), but decorated with a PID column and other kernel context that can't be seen on the wire. This turned out to be very complex without a stable tcp provider to use, especially correctly matching the TCP handshake packets, RST packets from closed port SYNs, and other TCP behavior. I should have stopped and gone back to the problems this would solve, primarily, identifying TCP sessions to their PID, and quantifying their workload. That could have been solved with tracing send and receive packets alone. Other objectives, like tracing closed port packets, should have been handled by separate scripts. Doing everything at once was ideal, but not practical at the time.</p>
<p>The DTrace book taught me discipline for creating short, simple, and useful scripts, that fit on half a textbook page. I think that's a better approach where possible, which may mean several small scripts for specific problems, instead of one long script. That will help with readability, maintenance, and testing.</p>
<h3>Mistake 5. The fbt Provider</h3>
<p>I suspected I was making this mistake at the time, and I was. The function boundary tracing (fbt) provider does dynamic tracing of kernel functions, arguments, return values, and data structures. While the observability is amazing, any script that is fbt-based is tied to a kernel version (the code it instruments), and may stop working after an upgrade or a patch. This happened with tcpsnoop.d and tcptop, complex scripts that instrumented many TCP internals. An excerpt:</p>
<pre>
/*
* TCP Fetch "port closed" ports
*/
fbt:ip:tcp_xchg:entry
/self->reset/
{
#if defined(_BIG_ENDIAN)
self->lport = (uint16_t)arg0;
self->fport = (uint16_t)arg1;
#else
[...]
</pre>
<p>The function this traces, tcp_xchg(), was removed in a kernel update. This broke tcpsnoop.d. The sad part is that this code was only necessary for tracing closed port packets (RST), and most of the time people weren't using tcpsnoop.d for that. See the earlier mistake. Had this been separate scripts, the breakage would be isolated, and, easier to fix and test.</p>
<p>I stopped creating fbt-based TCP scripts after tcpsnoop and tcptop, waiting for a stable DTrace tcp provider to be developed (which I ended up developing). I think it was useful to have them there, even though they usually didn't work, because it has been important to show that DTrace can do kernel-level TCP tracing. See my other post on <a href="http://dtrace.org/blogs/brendan/2013/05/27/the-greatest-tool-that-never-worked-har/">the value of broken tools</a>.</p>
<h3>Mistake 6. The syscall Provider</h3>
<p>I knew this mistake was theoretically possible, but never expected it would actually happen. The DTrace syscall provider is technically an unstable interface, due to the way it instruments the kernel trap table implementation. The DTrace manual mentioned the couple of places where this mattered, as probe names differed from syscall names. So any script that traces syscalls could break after a kernel upgrade, if the syscall trap table interface changed. Well, I'd expect syscalls to be added, but I didn't expect much changes to the existing names.</p>
<p>Oracle Solaris 11 went ahead and changed the syscall trap table <a href="http://dtrace.org/blogs/brendan/2011/11/09/solaris-11-dtrace-syscall-provider-changes/">significantly</a>. This was to slightly simplify the code (housekeeping), making it a little easier for the few kernel engineers who maintain it. A side effect is that it broke many DTrace one-liners and scripts that customers were using. Taking that into consideration, I think this change was a mistake. It's adding more complexity for customers to endure than the function it provides (see the Mythical Man Month). In <a href="http://dtrace.org/blogs/brendan/2011/11/09/solaris-11-dtrace-syscall-provider-changes/">one use case</a>, customers on Oracle Solaris 11 must include the birthday of an engineer in their scripts (a magic number) in order to have similar functionality as before. This is just madness.</p>
<p>Who's mistake is this ultimately? Everyone's. The DTrace syscall provider implementation was unstable to begin with, and while not a big problem at the time, you could argue that it should have been implemented stable from the very beginning. Perhaps I should not have used it so much, although without a stable alternative, much of the value of the DTraceToolkit would be lost. Avoiding it would also have made DTrace adoption more difficult, as tutorials usually begin by tracing the well-understood system calls. But the biggest mistake may be Oracle making these changes without also fixing the syscall provider. They did fix DTraceToolkit scripts and shipped them with Oracle Solaris 11, but much of the existing documentation for DTrace has not been fixed, nor is it easy to. All documentation needs to note that Oracle Solaris 11 is a special case, where the trap table differences are so large you do need to learn them. Fail.</p>
<h3>Mistake 7. Scripts That Build Scripts</h3>
<p>I shouldn't have written complex scripts that generate DTrace scripts. I did this once, with errinfo, a Perl program that builds and executes DTrace as a co-process. After finishing that I realized it was a bad idea, and did no more. It makes the script a difficult example to learn from and difficult to maintain. What I did do for many scripts (iosnoop, opensnoop, etc), was to break the program in two halves: the top half is a shell script for processing options, and the bottom half is the D script. This kept it simple, and maintenance wasn't too difficult.</p>
<h3>Mistake 8. The Names iosnoop and iotop</h3>
<p>I should have called these disksnoop and disktop, to make it clear that they are tracing physical I/O and not logical I/O (syscalls). It's confused a lot of people, and I'm sorry. I should have picked better names. iotop has since been ported to Linux (more than once), where the name has been kept, so I've indirectly confused those users as well.</p>
<h3>Mistake 9. Not Testing iosnoop With High Enough Load</h3>
<p>I didn't test iosnoop with a high enough disk IOPS rate. My development servers had a limited number of disks, where it ran fine. But on large scale servers, people reported "dynamic variable drops". The fix was to increase the dynvarsize tunable, which I have done in the next version (comes after the "switchrate" line):</p>
<pre>
/* boost the following if you get "dynamic variable drops" */
#pragma D option dynvarsize=16m
</pre>
<p>I could also change the key to the associative arrays to reduce overhead. It is currently the device and block number (this->dev, this->blk), but using the pre-translated message pointer (arg0) also works for many if not all OSes. The down side is that using arg0 is an unstable interface, which I'd rather avoid.</p>
<h3>Mistake 10. Using io:genunix::start</h3>
<p>I should not have used this probe at all, or, I should have added a comment to explain why I was. io:::start traces all disk I/O from the block device interface, and, NFS client I/O. Specifying "genunix" as the module name matched the disk I/O only, which is what I wanted, as NFS I/O was from a different module. But that's an unstable way to do it, and these scripts didn't work on other OSes that didn't have "genunix" as the module name. In later versions of my scripts, I removed the genunix, which means they also match NFS I/O.</p>
<p>I explained this in the DTrace book, and mentioned the stable fix: using io:::start with the predicate /args[1]->dev_ name != "nfs"/.
<h3>Mistake 11. Using Global Associative Arrays</h3>
<p>These can become corrupted, and did for the cputimes script. This one in particular:
<pre>
/* determine the name for this thread */
program[cpu] = pid == 0 ? idle[cpu] ? "IDLE" : "KERNEL" :
OPT_all ? execname : "PROCESS";
</pre>
<p>That's saving a string to a global associative array, program[], which is keyed on the CPU id. Unlike aggregations or thread-local variables, these are <a href="http://dtrace.org/blogs/brendan/2011/11/25/dtrace-variable-types/">not multi-CPU safe</a>. cputimes would sometimes print strings that were garbled. Fortunately this wasn't subtle corruption, but was pretty obvious.</p>
<h3>Mistake 12. dapptrace</h3>
<p>I should have warned about the overheads of dapptrace more clearly. With the -a option, it traces every user-level function, which for some applications will significantly slow the target. Without -a, it traces just the application text segment, which, depending on the application, can also incur high overhead. I warned about it in the example file, but I should have warned about it in the script and man page as well.</p>
<p>Maybe I shouldn't have even written this tool. With applications, I'm usually tracing a specific function or group of functions, which minimizes the overhead.</p>
<h3>Mistake 13. Missing Units On Output</h3>
<p>I should have always put units on the output. For example, running iosnoop with completion times (-t, for the TIME column) and I/O delta times (start to done, -D, for the DELTA column):</p>
<pre>
# iosnoop -Dt
TIME DELTA UID PID D BLOCK SIZE COMM PATHNAME
633389165137 1903 0 1 W 749429752 4096 launchd ??/T/etilqs_sa...
633389165187 1461 0 1 W 385608448 4096 launchd ??/log/opendir...
633389165215 2018 0 1 W 749429656 8192 launchd ??/T/etilqs_sa...
[...]
</pre>
<p>Where they microseconds or milliseconds? I forget. While the units are documented in the script, the USAGE message, and the man page, it's really handy to have it on the output as well. The next version does this:</p>
<pre>
# iosnoop -Dt
TIME(us) DELTA(us) UID PID D BLOCK SIZE COMM PATHNAME
[...]
</pre>
<p>All scripts should include units on the output.</p>
<h3>Mistake 14. Capital Directories</h3>
<p>I'm not sure starting directories with a capital letter was a great idea. It helped differentiate the scripts from the directories, and listed directories first. eg:
<pre>
~/DTraceToolkit-0.99> ls
Apps Java Perl User install
Bin JavaScript Php Version iopattern
Code Kernel Proc Zones iosnoop
Cpu License Python dexplorer iotop
Disk Locks README dtruss opensnoop
Docs Man Ruby dvmstat procsystime
Examples Mem Shell errinfo rwsnoop
FS Misc Snippits execsnoop rwtop
Guide Net System hotkernel statsnoop
Include Notes Tcl hotuser
</pre>
<p>Well, that seems pretty useful. But then, so is aliasing ls to "ls -F", or the colorized version of ls. Those techniques won't list the directories first, but they will differentiate them. This may be a choice between having directories listed first and together, or, not needing to hit the shift key so much. Or, I rearrange things to not mix scripts with directories.</p>
<h3>Mistake 15. Not Organizing For Other OSes or Kernel Versions</h3>
<p>I didn't think much about organizing the toolkit to support multiple OSes, as those ports hadn't happened yet. I should have thought harder, since around the same time DTrace was being released as open source, and I honestly expected it to be on Linux by the end of 2005. All I did do was put the OS type and version at the top of the scripts (better than nothing). Organizing the hierarchy is a problem I need to wrestle with now.</p>
<h3>Mistake 16. Not Testing New Kernels</h3>
<p>I didn't have a test plan for handling numerous new kernel versions. In part because I didn't yet know how much of a problem it would be, but also because I didn't expect the DTraceToolkit to be around for that long (see next mistake). When I began writing the DTraceToolkit, there was only one kernel version to test. Soon there were two, three, and then more than I had servers at home to test on. Pretty quickly people were running the DTraceToolkit on Solaris versions that I hadn't tested at all, and I didn't have the spare equipment to test.</p>
<p>I did get a lot of help from <a href="http://www.nbl.fi/~nbl97/solaris/dtrace/">Stephan Parvu</a>, who automated testing and was able to quickly do a smoke test on several kernel versions and platforms. But as the years went by, Sun kept releasing new Solaris versions, and staying on top was hard. There have been 12 versions of Solaris 10 so far (last was Jan this year), on two different platforms (x86, SPARC), meaning there are 24 OS versions to test just for Solaris 10 coverage.</p>
<p>One problem was that I didn't have ready access to the different versions of Solaris to test: ideally, I'd want 24 online servers spanning every version of Solaris 10, and root logins. At one point someone at Sun was going to contribute a pool of test servers to the OpenSolaris project. Unfortunately, I had to sign the OpenSolaris contributor agreement before I could use them. I didn't, and that turned out later to be a good decision.</p>
<p>The other problem was that testing was very time consuming, and was most of the script development time. The smaller scripts would take about 2 hours to develop: 20 minutes to write the script, 20 minutes to write the example file and man page, and then 80 minutes to test it and fix bugs found during testing. Larger scripts took longer - some took weeks. It's easy to write a DTrace script that produces numbers, but it's harder produce accurate numbers. Testing often involved running a variety of "known workloads" and then seeing if the DTrace tool measured the workloads exactly, and investigating when it didn't.</p>
<p>Testing is even harder now than before, as there are multiple OSes to test.</p>
<h3>Mistake 17. Not Planning For Success</h3>
<p>I didn't anticipate the DTraceToolkit would be this successful, and that it would be used years later on multiple OSes. I half expected Sun to release some new DTrace-related thing, like a DTrace GUI, that would make the toolkit redundant. If I had expected it to last this long, I could have planned testing kernel versions and multiple OSes better. I guess this is a lesson for any project: what if it is really successful in the long term? What architectural choices should be made now, that won't be regretted later?
<h3>Mistake 18. Private Development and Testing</h3>
<p>I should have made the development version of the DTraceToolkit public, even though it contained partially tested scripts. This would have allowed others to easily participate in testing these scripts, accelerating their development. Some people did help test, but this required me to email around tarballs, when I could have just had a public URL. It didn't seem like a good idea in 2005 when I created the toolkit, but nowadays it is commonplace to find both stable and unstable versions of a project online.</p>
<h3>Mistake 19. Soliciting Contributions From Beginners</h3>
<p>I encouraged everyone to send me scripts, which was a mistake. Most of the submissions were from DTrace beginners, some noting "this is my first ever DTrace script". Many of these produced invalid or misleading metrics. Most of the beginners were also inexperienced programmers, who didn't test or know how to test. And many didn't follow the toolkit coding style, or any programming style. I was sympathetic as they did have good intentions, and they were trying to help my project. So I would try to explain how to improve or test their script. At one point I also wrote websites on <a href="http://www.solarisinternals.com/wiki/index.php/DTrace_Topics_Style">style</a>, <a href="http://www.solarisinternals.com/wiki/index.php/DTrace_Topics_Dos_Donts">dos and don'ts</a>, and <a href="http://www.solarisinternals.com/wiki/index.php/DTrace_Topics_Hints_Tips">hints & tips</a>.</p>
<p>Problem was, most beginners didn't really understand what I was talking about, unless I spent serious time explaining. Since they had already contributed their time, some felt that I was obligated to return the favor, and explain, and keep explaining, until they understood everything. This took hours, many more than it would take to write the scripts myself. And that time didn't pay off: once the beginner realized I was serious about accuracy and testing, or the real complexity of what they were doing, they usually lost interest.</p>
<p>A common misconception with beginners was that the hard work was in writing the script, and that any extra work, like testing and documentation, was minor and could just be done by me. Testing <i>is</i> the hard work, and is where most of the development time is spent.</p>
<p>I should have only encouraged experienced software engineers. Some of whom did send me scripts, which I included. These were often accurate, tested, and styled to begin with. They were usually created when the engineer was debugging a particular software issue, and had created a DTrace script to solve it after reading the source code and learning all the nuances.</p>
<h3>Mistake 20. Joining Sun</h3>
<p>I joined Sun in late 2006, and in terms of the DTraceToolkit, this was a mistake. My work on the DTraceToolkit mostly stopped, and not just because I was busier. It's not easy or short to explain why. There are a number of pros and cons when working on an open source project like this, and before Sun it seemed that the pros outweighed the cons. Pros included things like helping fellow sysadmins – which I really enjoyed – and creating tools that I'd use in my own career. But there were various cons too, and Sun added more, to the point where it was hard to justify volunteering my spare time for this project.</p>
<p>Just as an example – and only because it's the easiest to summarize – at one point Sun legal became involved, and investigated whether Sun could claim ownership of the DTraceToolkit. As an employee, I was now in the crosshairs of my own company's lawyers. It wasn't fun, and they tried, very hard, for months. Thanks again to those that helped eventually put a stop to that!</p>
<p>I don't have hard feelings towards the lawyers. In fact, I admire how hard they were working for Sun – just like I was, so on some level it was understandable. But other events were much less understandable, and much worse. Again, there's no easy or short way to explain it all, and the problems weren't limited to the DTraceToolkit - DTrace itself came under friendly fire at Sun as well.</p>
<p>Instead of working on the DTraceToolkit I've been writing books, which have similar incentives: I get to help other people in the industry, especially sysadmins, as well as create references for myself, which also include tools. I never stopped contributing - I changed the medium to do so.</p>
<h3>Learning From Mistakes</h3>
<p>I need to:</p>
<ul>
<li>Get the DTraceToolkit on github.</li>
<li>Organize it to support different OSes.</li>
<li>Update existing scripts and add missing scripts.</li>
<li>Split the DTraceToolkit into two: a small number of well-tested scripts for casual users, and a large number of less-tested scripts for performance engineers.</li>
</ul>
<p>And by less-tested I do mean probably broken. This is embracing the reality of dynamic tracing, which exposes interfaces that are not stable. That doesn't make the scripts useless, rather, these scripts need to be treated differently: they are examples of solving problems (<a href="http://dtrace.org/blogs/brendan/2013/05/27/the-greatest-tool-that-never-worked-har/">even if they don't work</a>), not ready-to-go tools.</p>
<p>The split can be either in the DTraceToolkit as subdirectories, or as separate toolkits. This addresses several of the mistakes listed above. I hope to follow up this post with another explaining how the split is done, which can refer to this post as background.</p>
<p>And the final lesson I should learn: I should stop writing books, to get my spare time back! (My colleagues wouldn't believe that for a second.) Ok, well, maybe I'll a break from writing <i>very long</i> books, how about that? :-)</p>
<br /></div>
Brendan Gregghttp://www.blogger.com/profile/03662900777558310512noreply@blogger.com4tag:blogger.com,1999:blog-13965874.post-45659861880807080152013-07-17T17:23:00.001-07:002013-07-17T18:27:32.947-07:00Systems Performance: Enterprise and the Cloud<div dir="ltr" style="text-align: left;" trbidi="on">
<p>I wrote another book, <a href="http://dtrace.org/blogs/brendan/2013/06/21/systems-performance-enterprise-and-the-cloud/">Systems Performance: Enterprise and the Cloud</a>. It will be out this year, and I'm really looking forward to it helping people improve performance of their systems. While the book has been drafted, I'm not sure how many final pages it will be, as it still needs to finish the composition and layout stages of publication.</p>
<p>All my spare time has been consumed with it (and before it, the previous DTrace book), which is why this blog has been quiet for many years.</p>
</div>
Brendan Gregghttp://www.blogger.com/profile/03662900777558310512noreply@blogger.com0tag:blogger.com,1999:blog-13965874.post-32172854763933855092011-12-31T23:24:00.000-08:002012-01-01T00:48:14.920-08:00Still WritingI haven't posted here in a few years, but I have been busy writing material, particularly for:<br /><ul><li>The <a href="http://dtrace.org/blogs/brendan/2011/02/23/dtrace-book-sample-chapter-file-systems/">DTrace book</a>: which has over 1,000 pages, much of it new content. This ate over a year of my spare time (and Jim's).</li><br /><li><a href="http://dtrace.org/blogs/brendan">dtrace.org/blogs/brendan</a>: my professional blog, for posts related to my work (although it's still mostly a spare time project). This was formerly on blogs.oracle.com/brendan, and before that on blogs.sun.com/brendan.</li></ul>For more of my recent writing, I've updated a summary under the Documentation section on my <a href="http://www.brendangregg.com/index.html#Documentation">homepage</a>, which includes posts from my dtrace.org blog and other places.<br /><br />I'll post more here in the coming year: this blog is for purely personal posts and projects (like the DTraceToolkit).Brendan Gregghttp://www.blogger.com/profile/03662900777558310512noreply@blogger.com0tag:blogger.com,1999:blog-13965874.post-53962440682186566682008-06-24T08:23:00.000-07:002008-06-24T08:41:24.304-07:00DTrace in New YorkBack in February I gave several DTrace talks in New York, including one at the New York OpenSolaris User Group meeting (<a href="http://opensolaris.org/os/project/nycosug/">NYCOSUG</a>). I used an updated slide deck and was asked to put the PDF on my blog; I think Isaac must have beaten me to it and put it <a href="http://mediacast.sun.com/users/unixmd/media/dtrace_NYC.pdf">here</a> (thanks!). I did intend to blog about this in case anyone was looking - sorry for the delay.<br /><br />The NYCOSUG had a good turn out and asked some great questions, allowing me to deviate from the prepared slides and cover other things of interest (which is the value of an in-person presentation.) After the presentation I realised there was one point I could have explained better, which would make an interesting blog post.<br /><br />I started with the following simple demos - the point of these is to build on something commonly understood (such as the behaviour of fork() and exec()), to introduce something new - DTrace.<br /><br />Tracing exec():<br /><pre># <b>dtrace -n 'syscall::exec*: { trace(execname); }'</b><br />dtrace: description 'syscall::exec*: ' matched 4 probes<br />CPU ID FUNCTION:NAME<br /> 0 98087 exece:entry bash <br /> 0 98088 exece:return ls <br />^C<br /></pre><br />In the above output, we traced an exece() system call - printing the current process name when we entered and returned from that system call. That process name changed from "bash" to "ls" (I executed "ls -l" in another window), which is what exec() does - replaces the current process image with another.<br /><br />While unsuprising, the significance is that we are able to dymanically trace this kernel activity whenever we would like, along with thousands of other kernel events. I could, for example, trace the time taken for exec() to execute; or the exit status when exec() returned along with the error code; I can also trace the internal operation of exec() with enough detail to fill hundreds of pages (I just counted 47396 lines of output when tracing every kernel function entry and return during exec()).<br /><br />Now to trace fork():<br /><pre># <b>dtrace -n 'syscall::fork*: { trace(pid); }'</b><br />dtrace: description 'syscall::fork*: ' matched 6 probes<br />CPU ID FUNCTION:NAME<br /> 0 98227 forksys:entry 87417<br /> 0 98228 forksys:return 90769<br /> 0 98228 forksys:return 87417<br />^C<br /></pre><br />The above system call has one entry and two returns - which is what we expect from the fork() family.<br /><br />Simple as this is, some interesting behaviour is already visible. Note that the parent returned before the child? On the fork() entry, the parent's PID is traced (87417), however the child PID (90769) appears first on return.<br /><br />It's possible that the output could be shuffled due to how DTrace uses per-CPU buffers to minimise performance impact; to double check, add a "timestamp" column and post sort:<br /><br /><pre># <b>dtrace -n 'syscall::fork*: { printf("%d %d", timestamp, pid); }' -o /tmp/out.dtrace</b><br />dtrace: description 'syscall::fork*: ' matched 6 probes<br /># <b>sort -n +3 /tmp/out.dtrace</b><br />CPU ID FUNCTION:NAME<br /> 1 98227 forksys:entry 268361844462135 87417<br /> 1 98228 forksys:return 268361844960455 90968<br /> 0 98228 forksys:return 268361844965924 87417<br /></pre><br />I asked the audience - why does the child return from fork() before the parent? I added that this was a very difficult question!<br /><br />Someone responded to say that this was how all operating systems worked - the parent process waits for the child to complete. I said I was just tracing fork() and the parent could be scheduled first - but deliberately isn't, and explained why. My answer left them confused - and it struck me afterwards that I should have explained this better.<br /><br />Consider this:<br /><pre># <b>dtrace -n 'syscall::fork*:,syscall::wait*: { trace(pid); }'</b><br />dtrace: description 'syscall::fork*:,syscall::wait*: ' matched 10 probes<br />CPU ID FUNCTION:NAME<br /> 0 98227 forksys:entry 87417<br /> 0 98228 forksys:return 91088<br /> 0 98228 forksys:return 87417<br /> 0 98163 waitsys:entry 87417<br /> 0 98164 waitsys:return 87417<br />^C<br /></pre><br />In the above output we can see both fork() and wait(), and we can discuss behaviour such as the parent process waiting for the child to complete (since I was in a shell running foreground commands.)<br /><br />But I was actually asking a much deeper question, that of thread scheduling immediately after the fork() system call, and <i>before</i> the parent has called wait(). Immediately after fork() you have two threads - which should go on-CPU first? The parent, so that it can get to wait() sooner and before the child may have exited? Or is there a reason to schedule the child first?<br /><br />As DTrace shows, the child is getting scheduled first, and the reason is one of performance. The source code explains in <a href="http://cvs.opensolaris.org/source/xref/onnv/onnv-gate/usr/src/uts/common/disp/ts.c">uts/common/disp/ts.c</a> :<br /><pre><br />/*<br /> * Child is placed at back of dispatcher queue and parent gives<br /> * up processor so that the child runs first after the fork.<br /> * This allows the child immediately execing to break the multiple<br /> * use of copy on write pages with no disk home. The parent will<br /> * get to steal them back rather than uselessly copying them.<br /> */<br />static void<br />ts_forkret(kthread_t *t, kthread_t *ct)<br /></pre><br />The fork() system call creates a clone of the parent, but rather than copy all memory pages to a new address space (which would add significant latency during process creation), Solaris bumps a reference counter on those memory pages to remember that two process refer to the same data. If one writes to a memory page later, this triggers a "copy on write" to create a private writable copy for that process. This means that expensive memory copies are only performed when needed - or if needed. Since a child process is likely to call exec(), it is likely to simply drop many existing memory references for the new process image, so copying those bytes would have been wasted cycles anyway.<br /><br />However if the parent is scheduled first - before the child has had a chance to exec() - then the parent may continue writing to its address space, triggering copy on writes. Then the child executes, calls exec(), and drops those newly copied pages anyway - which were copied in vain. To avoid this, the child is scheduled first - to call exec() as soon as possible, as described in the comment above.<br /><br />I learned about this behaviour when reading Solaris Internals 1st edition; but that was a time before DTrace and OpenSolaris. It's great that we all can now both read the code, and use DTrace to see it in operation.Brendan Gregghttp://www.blogger.com/profile/03662900777558310512noreply@blogger.com0tag:blogger.com,1999:blog-13965874.post-78088361072119249802008-02-18T01:19:00.002-08:002008-02-18T02:56:52.396-08:00DTraceToolkit in MacOS XApple included DTrace in MacOS X 10.5 (Leopard), released in October 2007. It's great to have DTrace available in MacOS X for its powerful application and kernel performance analysis. To think that there is now another kernel we can examine using DTrace is exciting - it's like discovering a new planet in the solar system.<br /><br />Apart from kernel analysis, DTrace also improves general usability by answering every day questions like: why are my disks rattling? or why does my browser keep hanging? Although, your average user may not write DTrace scripts to answer these questions themselves (it's better if they do), but instead use prewritten scripts.<br /><br />MacOS X includes a collection of DTrace scripts in <b>/usr/bin</b>, mostly from the <a href="http://www.brendangregg.com/dtrace.html#DTraceToolkit">DTraceToolkit</a>:<br /><pre>leopard# <b>grep -l DTrace /usr/bin/*</b><br />/usr/bin/bitesize.d<br />/usr/bin/cpuwalk.d<br />/usr/bin/creatbyproc.d<br />/usr/bin/dappprof<br />/usr/bin/dapptrace<br />/usr/bin/diskhits<br />/usr/bin/dispqlen.d<br />/usr/bin/dtruss<br />/usr/bin/errinfo<br />/usr/bin/execsnoop<br />/usr/bin/fddist<br />/usr/bin/filebyproc.d<br />/usr/bin/hotspot.d<br />/usr/bin/httpdstat.d<br />/usr/bin/iofile.d<br />/usr/bin/iofileb.d<br />/usr/bin/iopattern<br />/usr/bin/iopending<br />/usr/bin/iosnoop<br />/usr/bin/iotop<br />/usr/bin/kill.d<br />/usr/bin/lastwords<br />/usr/bin/loads.d<br />/usr/bin/newproc.d<br />/usr/bin/opensnoop<br />/usr/bin/pathopens.d<br />/usr/bin/pidpersec.d<br />/usr/bin/plockstat<br />/usr/bin/priclass.d<br />/usr/bin/pridist.d<br />/usr/bin/procsystime<br />/usr/bin/runocc.d<br />/usr/bin/rwbypid.d<br />/usr/bin/rwbytype.d<br />/usr/bin/rwsnoop<br />/usr/bin/sampleproc<br />/usr/bin/seeksize.d<br />/usr/bin/setuids.d<br />/usr/bin/sigdist.d<br />/usr/bin/syscallbypid.d<br />/usr/bin/syscallbyproc.d<br />/usr/bin/syscallbysysc.d<br />/usr/bin/topsyscall<br />/usr/bin/topsysproc<br />/usr/bin/weblatency.d<br /></pre><br />That's 44 DTraceToolkit scripts, plus plockstat from Solaris 10. While the DTraceToolkit now has over 200 scripts, it makes sense to pick out the most useful scripts for inclusion in /usr/bin. <br /><br />Popular scripts such as iosnoop can now be run by MacOS X users:<br /><pre><br />leopard# <b>iosnoop</b><br /> UID PID D BLOCK SIZE COMM PATHNAME<br /> 501 130 R 31987472 40960 Terminal ??/dyld/dyld_shared_cache_i386<br /> 501 130 R 7879952 8192 Terminal ??/SearchManager.nib/keyedobjects.nib<br /> 501 130 R 32132304 12288 Terminal ??/dyld/dyld_shared_cache_i386<br /> 501 130 R 32132528 4096 Terminal ??/dyld/dyld_shared_cache_i386<br /> 501 130 R 32047696 12288 Terminal ??/dyld/dyld_shared_cache_i386<br /> 501 130 R 32132592 4096 Terminal ??/dyld/dyld_shared_cache_i386<br /> 501 130 R 32131512 12288 Terminal ??/dyld/dyld_shared_cache_i386<br /> 501 130 R 32033296 12288 Terminal ??/dyld/dyld_shared_cache_i386<br /> 501 130 R 32044488 4096 Terminal ??/dyld/dyld_shared_cache_i386<br /> 501 130 R 32045064 4096 Terminal ??/dyld/dyld_shared_cache_i386<br /> 501 130 R 32131344 4096 Terminal ??/dyld/dyld_shared_cache_i386<br /> 501 130 R 32048680 16384 Terminal ??/dyld/dyld_shared_cache_i386<br /> 501 130 R 32132544 8192 Terminal ??/dyld/dyld_shared_cache_i386<br /> 501 130 R 32049296 12288 Terminal ??/dyld/dyld_shared_cache_i386<br /> -1 0 W 32482848 86016 kernel_task ??/vm/swapfile2<br /> -1 0 W 32483040 135168 kernel_task ??/vm/swapfile2<br /> 501 130 R 32044672 4096 Terminal ??/dyld/dyld_shared_cache_i386<br /> 501 130 R 32132656 12288 Terminal ??/dyld/dyld_shared_cache_i386<br />[...]<br /></pre><br />The man pages are conveniently included in <b>/usr/share/man</b>.<br /><br />I had been making preperations in the latest DTraceToolkit (0.99) for MacOS X DTrace, such as putting an "OS" field into the man pages and figuring out how to support different versions of the same script (tcpsnoop_snv, etc). Hopefully many scripts will run on both Solaris and MacOS X (especially if they use stable providers), however I expect there will be some that are specific to each. Now that QNX DTrace also exists, there is additional need for identifying OS specifics in the DTraceToolkit.<br /><br />It's been great news for DTrace, Sun and Apple - who have not only gained the best performance and debugging tool available, but also the existing DTrace community.Brendan Gregghttp://www.blogger.com/profile/03662900777558310512noreply@blogger.com1tag:blogger.com,1999:blog-13965874.post-9309776488970464762008-02-17T23:19:00.002-08:002008-02-18T00:32:39.723-08:00Browsable DTraceToolkit<a href="http://stefanparvu.blogspot.com/">Stefan Parvu</a> has created browsable HTML versions of the <a href="http://www.brendangregg.com/dtrace.html#DTraceToolkit">DTraceToolkit</a> on the <a href="http://www.nbl.fi/~nbl97/solaris/dtrace/dtt_testing.html">DTT test page</a>. See <a href="http://www.nbl.fi/~nbl97/solaris/dtrace/099html/dtt099.html">DTraceToolkit ver 0.99</a> to browse that version. <br /><br />A goal of the DTraceToolkit is to provide documented examples of DTrace scripting, in addition to what is available in the <a href="http://wikis.sun.com/display/DTrace/Documentation">DTrace Guide</a>. However these examples have been reaching a limited audience of those who download, unzip, and browse through the text files.<br /><br />Now that the DTraceToolkit is browsable online, its contents can be found by internet search engines. This should help people not only find examples of DTrace usage, but also solutions to some common observability problems.<br /><br />There have been a few other items of DTraceToolkit news which I'll blog about soon. Please excuse my infrequent blog postings - I've been busy since joining Sun on a particular project which consumes most of my spare time. It will be worth it, which should be clear once I can start posting about it on my <a href="http://blogs.sun.com/brendan">Sun blog</a>.Brendan Gregghttp://www.blogger.com/profile/03662900777558310512noreply@blogger.com0tag:blogger.com,1999:blog-13965874.post-40347697047651562082007-10-05T23:13:00.000-07:002007-10-06T21:07:08.745-07:00DTraceToolkit ver 0.99I've released <a href="http://www.opensolaris.org/os/community/dtrace/dtracetoolkit">DTraceToolkit ver 0.99</a> - a major release. If you haven't encountered it before, the DTraceToolkit is a collection of opensource scripts for:<br /><ul><li>solving various troubleshooting and performance issues</li><li>demonstrating what is possible with DTrace (seeing is believing)</li><li>learing DTrace by example</li></ul><br />The DTraceToolkit isn't Sun's standard set of Solaris performance tools, or, everything that DTrace can do. It is a handy collection of documented and tested DTrace tools that are useful in most situations. It's certainly not a substitute for learning DTrace properly and writing your own custom scripts - although it should help you do that by providing working examples of code. It is certainly better than not using DTrace at all, if you didn't have the time to learn DTrace.<br /><br /><b>DTraceToolkit 0.96</b><br /><br />18 months ago I released version 0.96 of the DTraceToolkit. The Bin directory, which contains symlinks to all the scripts, looked like this,<br /><br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjVMNxZJp8OyN26in6_M518gq9Y-Nl7-e3gBLBTPe0C71ff4geW4wq8bucTiQ5p-pppsegliwhN9q0bGc5D6fc1-0KYsu7xhK1AGQjUKoiw72ZCGKmg9F2oT-2CcChMWV2OJwKx/s1600-h/Bin096_01c.png"><img style="margin: 0px auto 10px; display: block; text-align: center; cursor: pointer;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjVMNxZJp8OyN26in6_M518gq9Y-Nl7-e3gBLBTPe0C71ff4geW4wq8bucTiQ5p-pppsegliwhN9q0bGc5D6fc1-0KYsu7xhK1AGQjUKoiw72ZCGKmg9F2oT-2CcChMWV2OJwKx/s400/Bin096_01c.png" alt="" id="BLOGGER_PHOTO_ID_5118121440330917794" border="0" /></a><br /><center><i>DTraceToolkit version 0.96</i></center><br />Click for a larger image. That collection of scripts provides various views of system behaviour and of the interaction of applications with the system. A general theme, if there was one, was that of easy system observability - illuminating system internals that were either difficult or impossible to see before. There were 104 scripts in that release - although that is just the tip of an iceberg.<br /><br /><b>DTraceToolkit 0.99</b><br /><br />In the last 18 months, DTrace has been expanding its horizons through the creation of numerous new DTrace providers. There are now providers for Java, JavaScript, Perl, Python, Php, Ruby, Shell and Tcl, and more on the way. The iceberg of system observability is now just one in a field of icebergs, with more rising above the surface as time goes by. It's about time the DTraceToolkit was updated to refelect the trajectory of DTrace itself - which isn't just about system observability - it is the observability of life, the universe, and everything.<br /><br />This new version of the DTraceToolkit does has a theme - that of programing language visibility, which covers several of the new DTrace providers that now exist. The Bin directory of the toolkit now contains 230 scripts, and looks like,<br /><br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjrtthgoIEE3HYWMFwu0I7kVZTZVpt6U8eP-g5nOlx9fOFh5RPPfWzqnPOdGwAhDLaAobR4e_oN8-0LdeEphpMckvHmYSgUQ15R0zn-vzzmmIHgRgi0UH-urF09RtDJ98WIX-y0/s1600-h/Bin099_01c.png"><img style="margin: 0px auto 10px; display: block; text-align: center; cursor: pointer;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjrtthgoIEE3HYWMFwu0I7kVZTZVpt6U8eP-g5nOlx9fOFh5RPPfWzqnPOdGwAhDLaAobR4e_oN8-0LdeEphpMckvHmYSgUQ15R0zn-vzzmmIHgRgi0UH-urF09RtDJ98WIX-y0/s400/Bin099_01c.png" alt="" id="BLOGGER_PHOTO_ID_5118130378157860786" border="0" /></a><br />Below I've highlighted most of the new contents, grouped by language that they trace,<br /><br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhbRI9du30FQRGAxWBeAYfBjeRs6_nFdFEF9BfwyCH_WkQbbC8bPwriYGLDfAvSjnCp0F6Q3DapPYJyJBgyjr45c_JREBZLw-xDFGg1haFmzzEVRTenQ5kOkVPuYo0LMU1O-67q/s1600-h/Bin099_04c.png"><img style="margin: 0px auto 10px; display: block; text-align: center; cursor: pointer;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhbRI9du30FQRGAxWBeAYfBjeRs6_nFdFEF9BfwyCH_WkQbbC8bPwriYGLDfAvSjnCp0F6Q3DapPYJyJBgyjr45c_JREBZLw-xDFGg1haFmzzEVRTenQ5kOkVPuYo0LMU1O-67q/s400/Bin099_04c.png" alt="" id="BLOGGER_PHOTO_ID_5118130910733805506" border="0" /></a><br /><center><i>DTraceToolkit version 0.99</i></center><br />I've placed language scripts in their own subdirectories, each of which include a "Readme" file to document suggestions for where to find the provider and how to get it installed and working. Some of these providers do require downloading of source, patching, and compiling, which may take you some time. Both C and C++ are supported by numerous scripts, but don't have a prefix to group them like the other languages do; I hope to fix this in the next release, and actually have C and C++ subdirectories.<br /><br />DTraceToolkit 0.99 change summary:<br /><ul><li>New script categories: Java, JavaScript, Perl, Php, Python, Ruby, Shell and Tcl</li><li>Many new scripts (script count has doubled)</li><li>Many new example files (since there is one per script)</li><li>A new Code directory for sample target code (used by the example files)</li><li>Several bug fixes, numerous updates</li><li>Updated versions of tcpsnoop/tcptop for OpenSolaris/Solaris Nevada circa late 2007</li></ul><br />For each language, there is about a dozen scripts to provide:<br /><ul><li>general observability (such as function and object counts, function flows)</li><li>performance observability (on-cpu/elapsed times, function inclusive/exclusive times, memory allocation)</li><li>some paths for deeper analysis (syscall and library tracing, stacks, etc)</li></ul><br /><b>Screenshots</b><br /><br />Now for some examples. I'll demonstrate PHP, but the following applies to all the new languages supported in the toolkit (look in the Example directory for the language that interests you). Here is a sample PHP program that appears in the toolkit as Code/Php/func_abc.php,<br /><pre>/opt/DTT> <b>cat -n Code/Php/func_abc.php </b><br /> 1 <?php<br /> 2 function func_c()<br /> 3 {<br /> 4 echo "Function C\n";<br /> 5 sleep(1);<br /> 6 }<br /> 7<br /> 8 function func_b()<br /> 9 {<br /> 10 echo "Function B\n";<br /> 11 sleep(1);<br /> 12 func_c();<br /> 13 }<br /> 14<br /> 15 function func_a()<br /> 16 {<br /> 17 echo "Function A\n";<br /> 18 sleep(1);<br /> 19 func_b();<br /> 20 }<br /> 21<br /> 22 func_a();<br /> 23 ?></pre><br />For general observability there are scripts such as php_funccalls.php, for counting function calls,<br /><pre># <b>php_funccalls.d </b><br />Tracing... Hit Ctrl-C to end.<br />^C<br />FILE FUNC CALLS<br />func_abc.php func_a 1<br />func_abc.php func_b 1<br />func_abc.php func_c 1<br />func_abc.php sleep 3<br /></pre><br />The above output showed that sleep() was called three times, and the user defined functions were each called once - as we would have assumed from the source. With DTrace you can check your assumptions through tracing, which is often a useful excercise. I've solved many performance issues by examining areas that seem almost too obvious to bother checking.<br /><br />The following script is both for general and performance observability. It prints function flow along with delta times,<br /><pre># <b>php_flowtime.d</b><br /> C TIME(us) FILE DELTA(us) -- FUNC<br /> 0 3646108339057 func_abc.php 9 -> func_a<br /> 0 3646108339090 func_abc.php 32 -> sleep<br /> 0 3646109341043 func_abc.php 1001953 <- sleep<br /> 0 3646109341074 func_abc.php 31 -> func_b<br /> 0 3646109341098 func_abc.php 23 -> sleep<br /> 0 3646110350712 func_abc.php 1009614 <- sleep<br /> 0 3646110350745 func_abc.php 32 -> func_c<br /> 0 3646110350768 func_abc.php 23 -> sleep<br /> 0 3646111362323 func_abc.php 1011554 <- sleep<br /> 0 3646111362351 func_abc.php 27 <- func_c<br /> 0 3646111362361 func_abc.php 10 <- func_b<br /> 0 3646111362370 func_abc.php 9 <- func_a<br />^C </pre><br />Delta times, in this case, show the time from the line on which they appear to the previous line. The large delta times visible show that from when the sleep() function completed to when it began, took around 1.0 seconds of elapsed time. As we'd expect from the source. Both CPU (C) and time since boot (TIME(us)) are printed in case the output is shuffled on multi-CPU systems, and requires post sorting.<br /><br />A more consise way to examine elapsed times may be php_calltime.d, a script that can be useful for performance analysis,<br /><pre># <b>php_calltime.d</b><br />Tracing... Hit Ctrl-C to end.<br />^C<br /><br />Count,<br />FILE TYPE NAME COUNT<br />func_abc.php func func_a 1<br />func_abc.php func func_b 1<br />func_abc.php func func_c 1<br />func_abc.php func sleep 3<br />- total - 6<br /><br />Exclusive function elapsed times (us),<br />FILE TYPE NAME TOTAL<br />func_abc.php func func_c 330<br />func_abc.php func func_b 367<br />func_abc.php func func_a 418<br />func_abc.php func sleep 3025644<br />- total - 3026761<br /><br />Inclusive function elapsed times (us),<br />FILE TYPE NAME TOTAL<br />func_abc.php func func_c 1010119<br />func_abc.php func func_b 2020118<br />func_abc.php func sleep 3025644<br />func_abc.php func func_a 3026761</pre><br />The exclusive function elapsed times show which functions cused the latency - which identifies sleep() clocking a total of 3.0 seconds (as it was called three times). The inclusive times show which higher level functions the latency occured in, the largest being func_a() since every other function were called within it. There isn't a total line printed for inclusive times - since it wouldn't make much sense to do so (that column sums to 9 seconds, which is more confusing than of actual use).<br /><br />As a starting point for deeper analysis, the following script adds syscall tracing to the PHP function flows, and adds terminal escape colours (see Notes/ALLcolors_notes.txt for notes on using colors with DTrace),<br /><pre># <b>php_syscolors.d</b><br />C PID/TID DELTA(us) FILE:LINE TYPE -- NAME<br /><span style="color: rgb(160, 0, 160);">0 18426/1 8 func_abc.php:22 func -> func_a</span><br /><span style="color: rgb(160, 0, 160);">0 18426/1 41 func_abc.php:18 func -> sleep</span><br /><span style="color: rgb(0, 160, 0);">0 18426/1 15 ":- syscall -> nanosleep</span><br /><span style="color: rgb(0, 160, 0);">0 18426/1 1008700 ":- syscall <- nanosleep</span><br /><span style="color: rgb(160, 0, 160);">0 18426/1 30 func_abc.php:18 func <- sleep</span><br /><span style="color: rgb(160, 0, 160);">0 18426/1 42 func_abc.php:19 func -> func_b</span><br /><span style="color: rgb(160, 0, 160);">0 18426/1 28 func_abc.php:11 func -> sleep</span><br /><span style="color: rgb(0, 160, 0);">0 18426/1 14 ":- syscall -> nanosleep</span><br /><span style="color: rgb(0, 160, 0);">0 18426/1 1010083 ":- syscall <- nanosleep</span><br /><span style="color: rgb(160, 0, 160);">0 18426/1 29 func_abc.php:11 func <- sleep</span><br /><span style="color: rgb(160, 0, 160);">0 18426/1 43 func_abc.php:12 func -> func_c</span><br /><span style="color: rgb(160, 0, 160);">0 18426/1 28 func_abc.php:5 func -> sleep</span><br /><span style="color: rgb(0, 160, 0);">0 18426/1 14 ":- syscall -> nanosleep</span><br /><span style="color: rgb(0, 160, 0);">0 18426/1 1009794 ":- syscall <- nanosleep</span><br /><span style="color: rgb(160, 0, 160);">0 18426/1 28 func_abc.php:5 func <- sleep</span><br /><span style="color: rgb(160, 0, 160);">0 18426/1 34 func_abc.php:6 func <- func_c</span><br /><span style="color: rgb(160, 0, 160);">0 18426/1 18 func_abc.php:13 func <- func_b</span><br /><span style="color: rgb(160, 0, 160);">0 18426/1 17 func_abc.php:20 func <- func_a</span><br /><span style="color: rgb(0, 160, 0);">0 18426/1 21 ":- syscall -> fchdir</span><br /><span style="color: rgb(0, 160, 0);">0 18426/1 19 ":- syscall <- fchdir</span><br /><span style="color: rgb(0, 160, 0);">0 18426/1 9 ":- syscall -> close</span><br /><span style="color: rgb(0, 160, 0);">0 18426/1 13 ":- syscall <- close</span><br /><span style="color: rgb(0, 160, 0);">0 18426/1 35 ":- syscall -> semsys</span><br /><span style="color: rgb(0, 160, 0);">0 18426/1 12 ":- syscall <- semsys</span><br /><span style="color: rgb(0, 160, 0);">0 18426/1 7 ":- syscall -> semsys</span><br /><span style="color: rgb(0, 160, 0);">0 18426/1 7 ":- syscall <- semsys</span><br /><span style="color: rgb(0, 160, 0);">0 18426/1 66 ":- syscall -> setitimer</span><br /><span style="color: rgb(0, 160, 0);">0 18426/1 8 ":- syscall <- setitimer</span><br /><span style="color: rgb(0, 160, 0);">0 18426/1 39 ":- syscall -> read</span><br /><span style="color: rgb(0, 160, 0);">0 18426/1 14 ":- syscall <- read</span><br /><span style="color: rgb(0, 160, 0);">0 18426/1 11 ":- syscall -> writev</span><br /><span style="color: rgb(0, 160, 0);">0 18426/1 22 ":- syscall <- writev</span><br /><span style="color: rgb(0, 160, 0);">0 18426/1 23 ":- syscall -> write</span><br /><span style="color: rgb(0, 160, 0);">0 18426/1 110 ":- syscall <- write</span><br /><span style="color: rgb(0, 160, 0);">0 18426/1 61 ":- syscall -> pollsys</span></pre><br />From that output we can see that PHP's sleep() is implemented by the syscall nanosleep(), and that it appears that writes are buffered and written later.<br /><br /><b>Understanding the language scripts</b><br /><br />The scripts are potentially confusing for a couple of reasons; understanding these will help explain how best to use these scripts.<br /><br />Firstly, some of the scripts print out an event "TYPE" field (eg, "syscall" or "func"), and yet the script only traces one type of event (eg, "func"). Why print a field that will always contain the same data? Sounds like it might be a copy-n-paste error. This is in fact deliberate so that the scripts form better templates for expansion, and was something I learnt while using the scripts to analyse a variety of problems. I would frequently have a script that measured functions and wanted to add syscalls, library calls or kernel events, to customise that script for a particular problem. When adding these other event types, it was much easier (often trivial) if the script already had a "TYPE" field. So now most of them do.<br /><br />Also, there are variants of scripts that provide very similar tracing at different depths of detail. Why not one script that does everything, with the most detail? Here I'll use the Tcl provider to explain why,<br /><br />The tcl_proccalls.d script counts procedure calls, here it is tracing the Code/Tcl/func_slow.tcl program in the toolkit, which calls loops within functions:<br /><pre><b># tcl_proccalls.d</b><br />Tracing... Hit Ctrl-C to end.<br />^C<br /> PID COUNT PROCEDURE<br />183083 1 func_a<br />183083 1 func_b<br />183083 1 func_c<br />183083 1 tclInit</pre><br />Ok, that's a very high level view of Tcl activity. There is also tcl_calls.d to count Tcl commands as well as procedures:<br /><pre># <b>tcl_calls.d</b><br />Tracing... Hit Ctrl-C to end.<br />^C<br /> PID TYPE NAME COUNT<br />[...]<br />183086 proc func_a 1<br />183086 proc func_b 1<br />183086 proc func_c 1<br />183086 proc tclInit 1<br />[...]<br />183086 cmd if 8<br />183086 cmd info 11<br />183086 cmd file 12<br />183086 cmd proc 12</pre><br />(Output truncated). Now we have a deeper view of Tcl operation. Finally there is tcl_ins.d to count Tcl instructions:<br /><pre># <b>tcl_ins.d</b><br />Tracing... Hit Ctrl-C to end.<br />^C<br /> PID TYPE NAME COUNT<br />[...]<br />16005 inst jump1 14<br />16005 inst pop 18<br />16005 inst invokeStk1 53<br />16005 inst add 600000<br />16005 inst concat1 600000<br />16005 inst exprStk 600000<br />16005 inst lt 600007<br />16005 inst storeScalar1 600016<br />16005 inst done 600021<br />16005 inst loadScalar1 1200020<br />16005 inst push1 4200193</pre><br />There are such large counts since this script is tracing the low level instructions used to execute the loops in this code.<br /><br />So the question is: why three scripts and not one that counts everything? Why not tcl_megacount.d? There are at least three reasons behind this.<br /><br />1. You don't pull out a microscope if you are looking for your umbrella. Tracing everything can really bog you down in detail, when higher level views are frequently sufficient for solving issues. It's a matter of the right tool for the job. Try the higher level scripts first, then dig deeper as needed.<br /><br />2. Measuring at such a low level can have a significant performance impact. I'll demonstrate this by using <tt>ptime</tt> as a easy (but rough) measurement of run time for the Code/Tcl/func_slow.tcl sample program.<br /><br />Here is func_slow.tcl running with no DTrace enabled as a baseline:<br /><pre># <b>ptime ./func_slow.tcl > /dev/null</b><br /><br />real 3.306<br />user 3.276<br />sys 0.005<br /></pre><br />Now while tcl_proccalls.d is tracing procedure calls:<br /><pre># <b>ptime ./func_slow.tcl > /dev/null</b><br /><br />real 3.311<br />user 3.270<br />sys 0.006<br /></pre><br />Now while tcl_calls.d is tracing both Tcl procedures and commands:<br /><pre># <b> ptime ./func_slow.tcl > /dev/null</b><br /><br />real 3.313<br />user 3.283<br />sys 0.006<br /></pre><br />Finally with tcl_ins.d tracing Tcl instructions:<br /><pre># <b>ptime ./func_slow.tcl > /dev/null</b><br /><br />real 20.438<br />user 20.278<br />sys 0.010<br /></pre><br />Ow! Our target application is running 7 times slower? This isn't supposed to happen, is there something wrong with DTrace or the Tcl provider? Well, there isn't, the problem is that of tracing 9 million events during a 3 second program. The overheads of DTrace are miniscule, however if you multiply them (or anything) by ridiculously large numbers they'll eventually become measurable. 9 million events is a lot of events - and you wouldn't normally care about tracing Tcl instructions anyway. It's great to have this capability, if it is ever needed.<br /><br />3. Example programs. Writing mega powerful DTrace scripts often means mega confusing scripts to read. People use the DTraceToolkit as a collection of example code to learn from, so keeping things simple if possible, should help.<br /><br /><b>Documentation</b><br /><br />It usually takes more time to document a script than it takes to write it. I feel strongly about creating quality IT documentation, so I'm reluctant to skip this step. The strategy used in the toolkit is, by subdirectory name,<br /><ul><li>Scripts - these have a standard header plus inline comments.</li><li>Examples - this is a directory of example files that demonstrate each script, providing screenshots and interpretations of the output. Particular attention has been made to explain relevant details, especially caveats, no matter how obvious they may seem.</li><li>Notes - this is a directory of files that comment on a variety of topics that are relevant to multiple scripts. There are many more notes files in this toolkit release.</li><li>Man pages - there is a directory for these, which are useful for looking up field definitions and units. To be honest, I frequently use the Examples directory for reminders on what each tool does rather than the Man pages.</li></ul><br />Since there were over 100 new scripts in this release, over 100 new example files needed to be written. Fortunately I had help from Claire Black (my wife), who as an ex-SysAdmin and Solaris Instructor had an excellent background for what people needed to know and how to express it. Unfortunately for her, she only had about 4 days from when I completed these scripts to when the documentation was due for inclusion on the next OpenSolaris Starter Kit disc. Despite the time constraint she has done a great job, and there is always the next release for her to spend more time fleshing out the examples further. I'm also aiming to collect more real-world example outputs for the next release (eg, Mediawiki analysis for the PHP scripts).<br /><br /><b>Testing</b><br /><br />Stefan Parvu has helped out again by running every script on a variety of platforms, which is especially important now that I don't have a SPARC server at the moment. He has posted the results on the <a href="http://www.nbl.fi/%7Enbl97/solaris/dtrace/dtt_testing.html">DTraceToolkit Testing Room</a>, and did find a few bugs which I fixed before release.<br /><br /><b>Personal</b><br /><br />It has been about 18 months since the last release. In case anyone was wondering, there are several reasons why, briefly: I moved to the US, began a new job in Sun engineering, left most of my development servers behind in Australia, brought the DTraceToolkit dev environment over on a SPARC UFS harddisk (which I still can't read), and got hung up on what to do with the fragile tcp* scripts. The main reason would be having less spare time as I learnt the ways of Sun engineering.<br /><br />I also have a Sun blog now, <a href="http://blogs.sun.com/brendan">here</a>. I'm not announcing the DTraceToolkit there since it isn't a Sun project; it's a spare time project by myself and others of the OpenSolaris community.<br /><br /><b>Future</b><br /><br />The theme for the next release may be storage: I'd like to spend some time seeing what can be done for ZFS, for iSCSI (using the new provider), and NFS.Brendan Gregghttp://www.blogger.com/profile/03662900777558310512noreply@blogger.com2tag:blogger.com,1999:blog-13965874.post-1166090942986605912006-12-14T00:00:00.000-08:002007-03-20T19:48:46.062-07:00Updates from CaliforniaSince I've been quiet for a while on this blog and on <a href="http://www.opensolaris.org">OpenSolaris</a> mailing lists, here is a quick update about myself and my projects.<br /><br />I got married, ditched my old <a href="http://www.brendangregg.com/Perf/solarismetrics.html">car</a>, left the instructing/consulting scene in Australia/APAC, and moved to California to work as a developer at Sun. I brought along whatever I could fit in airline luggage, which included a laptop, hard disks, some books, clothes, and my fencing gear. Since then, I've been busy getting up to speed with some interesting projects, some of which I've discussed on my Sun <a href="http://blogs.sun.com/brendan">blog</a>.<br /><br />OpenSolaris projects such as the <a href="http://www.opensolaris.org/os/community/dtrace/dtracetoolkit">DTraceToolkit</a> will remain OpenSolaris projects; and other open source projects such as <a href="http://www.brendangregg.com/chaosreader.html">Chaosreader</a> will of course remain open source. I'll post updates about them to this blog and will continue working on them as I always have done - as a spare time hobby.<br /><br />Progress on these other projects has been on hold since I left Australia and powered down my array of test and development servers (and sadly powered down "mars", my main server which had an uptime of over 400 days). I began rebuilding my development environment last night, when I powered on the disk containing the DTraceToolkit build files for the first time in three months. I'm looking forward to adding some more scripts, and more documentation to guide people through the toolkit.<br /><br />And finally, living in the US has been great - however the following items were quite difficult to find: <ol><li>cricket coverage</li><li>authentic ginger beer</li></ol> I can now listen to Star 101.3 again, but I've lost access to both the ABC and BBC's online radio coverage of the Ashes Cricket series - which seems to be blocked from the US!Brendan Gregghttp://www.blogger.com/profile/03662900777558310512noreply@blogger.com0tag:blogger.com,1999:blog-13965874.post-1155532916317035652006-08-13T22:03:00.000-07:002006-08-13T22:23:13.796-07:00No more Star 101.3If you are in Australia, that is.<br /><br />Sometime last year I went looking for a decent Internet radio station, as I'm often online on my laptop, but not ofter near radios or reliable reception. I found <a href="http://www.star1013fm.com/listenlive.html">Star 101.3 FM</a>, and it was good. It happens to be based in the US.<br /><br />My day would often begin with info bytes from John Tesh followed by hours of music (ok, sometimes they play the same music ;-). When I'd hear "Don Bleu in the morning", I'd know I'd stayed up a bit too late!<br /><br />Recently Star 101.3 pulled the plug for international listeners "due to licensing restrictions". Ok, I can understand that may well be the case, and I'm glad it lasted as long as it did. If you are in the US you can still listen.Brendan Gregghttp://www.blogger.com/profile/03662900777558310512noreply@blogger.com2tag:blogger.com,1999:blog-13965874.post-1155456071048511012006-08-13T00:38:00.000-07:002006-08-13T01:01:11.060-07:00Brendan Clones #2Several months ago several of the Brendan clones were <a href="http://bdgregg.blogspot.com/2006/01/brendan-clones-recently-james-dickens.html">photographed</a> while working on the <a href="http://www.opensolaris.org/os/community/dtrace/dtracetoolkit">DTraceToolkit</a>.<br /><br />More recently they were photographed again, this time working on <a href="http://www.sun.com/books/catalog/solaris_internals.xml">Solaris Internals</a> and <a href="http://www.sun.com/books/catalog/solaris_perf_tools.xml">Solaris Performance and Tools</a>. For the first time all of the Brendan clones were involved on the one project, with most of them pictured here. Click for a larger photo:<br /><br /><a href="http://www.brendangregg.com/Photos/Misc/brendan_clones02.jpg"><img src="http://www.brendangregg.com/Photos/Misc/sml_brendan_clones02.jpg" border="0" /></a><br /><br />Sleeping Brendan is busy sleeping, Engineer Brendan is studying Solaris Internals 1st edition, and Master Brendan is looking on. The remaining Brendans are meticulously checking facts, reading source code, writing DTrace scripts and testing them.<br /><br />Of course, photographer Brendan is behind the camera, taking the photo.Brendan Gregghttp://www.blogger.com/profile/03662900777558310512noreply@blogger.com0tag:blogger.com,1999:blog-13965874.post-1155316229209747492006-08-11T07:28:00.000-07:002006-08-11T10:15:03.166-07:00How much CPU does a down interface chew?While teaching a DTrace class in Sydney this week I ran an ad-lib demo that took a mysterous and suprising turn. I think you'll find this quite interesting.<br /><br />One of my favourite DTrace demos begins by asking how much CPU a busy network interface will consume. About 2 in 10 people will know the answer - a lot! But how did you measure this on older Solaris? After all, if it really is "a lot", then we really want a value! 20% of a CPU? 50%?<br /><br />I was demonstrating on an Ultra 10, with a 333 MHz CPU and a 100 Mbit/sec interface (hme0). Even with the numerous TCP/IP performance enhancements in Solaris 10, driving 100 Mbit/sec on an Ultra 10 will still hurt.<br /><br />From another host, I sent a large file over the network to drive the interface at capacity. I thought I'd start by showing how busy the network interface was by using the sysperfstat tool from the <a href="http://www.brendangregg.com/k9toolkit.html">K9Toolkit</a> (I usually use nicstat, I thought this would be a nice change). sysperfstat ran for a few seconds, then I began the bulk network transfer:<br /><pre>$ /brendan/K9Toolkit/sysperfstat 1<br /> Time %CPU %Mem %Disk %Net CPU Mem Disk Net<br />19:24:27 5.24 33.52 0.52 0.07 0.05 0.00 0.01 0.00<br />19:24:28 7.35 33.53 1.29 0.00 0.00 0.00 0.00 0.00<br />19:24:29 7.48 33.53 0.00 0.00 0.00 0.00 0.00 0.00<br />19:24:30 39.65 33.79 1.27 51.27 3.69 0.00 0.00 0.00<br />19:24:32 65.67 33.79 0.00 173.14 1.28 0.00 0.00 0.00<br />19:24:34 68.35 33.79 0.00 173.03 0.00 0.00 0.00 0.00<br />19:24:37 80.12 33.79 0.00 176.33 0.82 0.00 0.00 0.00<br />19:24:39 76.97 33.79 0.00 178.12 0.43 0.00 0.00 0.00<br />19:24:41 76.08 33.79 1.61 164.82 1.37 0.00 0.00 0.00<br />19:24:44 80.18 33.79 0.00 177.68 0.38 0.00 0.00 0.00 </pre><br />Oh dear! I was expecting %Net to reach around 90%, not over 170%! <br /><br />Both the Net and Disk percentages are in terms of a single interface or disk, as from experience this has shown to be more useful than a global percentage (which CPU and Mem use). 170% could indicate that I have two network interfaces running at around 85%. But why? I had only initiated one network transfer to the hme0 address.<br /><br />... The system had become so heavily loaded that sysperfstat's interval of 1 second became 2 and sometimes 3 seconds. Good thing I never assumed sleep(1) would actually sleep for 1 second only - sysperfstat uses delta times from kstat measurements.<br /><br />Demonstrations are much more exciting when even you don't know of the outcome. So why did we hit 170% utilisation? Maybe another network interface was doing traffic that I didn't know about:<br /><pre># ifconfig -a<br />lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1<br /> inet 127.0.0.1 netmask ff000000<br />hme0: flags=1000863<UP,BROADCAST,NOTRAILERS,RUNNING,MULTICAST,IPv4> mtu 1500 index 2<br /> inet 192.168.1.13 netmask ffffff00 broadcast 192.168.1.255<br /> ether 8:0:20:b0:7a:18<br />qfe0: flags=1000862<BROADCAST,NOTRAILERS,RUNNING,MULTICAST,IPv4> mtu 1500 index 3<br /> inet 10.1.2.3 netmask ff000000 broadcast 10.255.255.255<br /> ether 8:0:20:b0:7a:18<br />#</pre><br />No answers there. <code>hme0</code> is "UP", <code>qfe0</code> is both down and has an address class that we aren't using anywhere at this site. So qfe0 can't be doing anything, right?<br /><pre>$ /brendan/K9Toolkit/nicstat 1<br /> Time Int rKb/s wKb/s rPk/s wPk/s rAvs wAvs %Util Sat<br />19:25:34 qfe0 3.77 0.00 2.98 0.00 1292.8 42.00 0.03 0.00<br />19:25:34 hme0 4.69 0.42 3.77 5.92 1272.9 72.52 0.04 0.00<br /> Time Int rKb/s wKb/s rPk/s wPk/s rAvs wAvs %Util Sat<br />19:25:36 qfe0 10435.6 0.00 8642.5 0.00 1236.4 0.00 85.49 0.00<br />19:25:36 hme0 10394.1 818.6 8601.4 15524.7 1237.4 54.00 91.86 0.00<br /> Time Int rKb/s wKb/s rPk/s wPk/s rAvs wAvs %Util Sat<br />19:25:39 qfe0 10469.2 0.00 8744.1 0.00 1226.0 0.00 85.76 0.00<br />19:25:39 hme0 10451.4 828.3 8732.5 15708.5 1225.5 54.00 92.40 0.00 </pre><br />Woah - this "down" <code>qfe0</code> with a dummy address is 85% busy - somthing really strange is going on! <br /><br />I had began this demo to show %CPU for busy interfaces, which leads us to run the the DTrace based "intrstat" tool - new to Solaris 10. This shows us how much CPU our network interface interrupts are consuming, and now I had a renewed interest in what it had to say:<br /><pre># intrstat 1<br /><br /> device | cpu0 %tim<br />-------------+---------------<br /> hme#0 | 1232 32.8<br /> m64#0 | 0 0.0<br /> qfe#0 | 1140 37.8<br /> su#0 | 0 0.0<br /> su#1 | 0 0.0<br /> uata#0 | 0 0.0<br /><br /> device | cpu0 %tim<br />-------------+---------------<br /> hme#0 | 1233 32.9<br /> m64#0 | 0 0.0<br /> qfe#0 | 1138 37.8<br /> su#0 | 0 0.0<br /> su#1 | 0 0.0<br /> uata#0 | 1 0.0<br />...</pre><br />hme0 is consuming 33% of CPU0, quite a substantial amount - considering this is just a 100 Mbit/sec interface. That's where the planned demo would have ended.<br /><br />However qfe0 is consuming 37.8% of CPU0 - this is really frightening! Remember, I am sending traffic to hme0's address, and qfe0 is not marked "UP". This is a dead interface with a dummy address - and it is consuming a LOT of CPU! (perhaps it is talking to other dead interfaces? ;-)<br /><br />I began thinking outside the box, and <a href="http://blogs.sun.com/doc">Scott</a> was around to help. We found that the room was connected to a hub, both hme0 and qfe0 were connected to this same hub, and they both had the same ethernet address. So while qfe0 eventually rejected the packets that it could see, it had to walk through much kernel code to get to that point - consuming CPU cycles and incrementing kstats. (a clue from the nicstat output was that qfe0 was receiving but not sending - so it wasn't taking part in a connection).<br /><br />I unplugged qfe0 and the problem vanished. Easy. However associating CPU load with a down interface is far from easy!<br /><br />...<br /><br />As an aside, I also ask how we measure how <i>utilised</i> the network interfaces are. Very few people have been able to answer this - and it is a topic I hope chapter 7 of this <a href="http://www.sun.com/books/catalog/solaris_perf_tools.xml">book</a> will increase awareness for. (eg, nicstat or sysperfstat from the <a href="http://www.brendangregg.com">K9Toolkit</a>).Brendan Gregghttp://www.blogger.com/profile/03662900777558310512noreply@blogger.com4tag:blogger.com,1999:blog-13965874.post-1155057121771212452006-08-08T10:04:00.000-07:002006-08-08T10:13:18.830-07:00Solaris Performance and ToolsBoth <a href="http://www.sun.com/books/catalog/solaris_internals.xml">Solaris Internals 2nd Edition</a> and <a href="http://www.sun.com/books/catalog/solaris_perf_tools.xml">Solaris Performance and Tools</a> are available to purchase, and copies have now began arriving around the World. They are both incredible books, covering an enormous range of topics with details not found anywhere else.<br /><table border=0><tr><td><br /> <a href="http://www.amazon.com/gp/product/0131482092" border=0><br /> <img src="http://users.tpg.com.au/adsln4yb/Photos/Misc/0131482092_sml.jpg"><br /> </a><br /></td><td><br /> <a href="http://www.amazon.com/gp/product/0131568191" border=0><br /> <img src="http://users.tpg.com.au/adsln4yb/Photos/Misc/0131568191_sml.jpg"><br /> </a><br /></td></tr></table><br />During my years working in the field of performance, I've taught numerous performance classes for Sun Microsystems where I gained both the experience and understanding of how to best teach performance. I have also written numerous open source programs for performance monitoring, such as tools in the <a href="http://www.opensolaris.org/os/community/dtrace/dtracetoolkit">DTraceToolkit</a> and the <a href="http://www.brendangregg.com/k9toolkit.html">K9Toolkit</a>. It was a privilege to share the best of my performance knowledge and teaching experience by co-authoring Solaris Performance and Tools with <a href="http://blogs.sun.com/rmc">Richard</a> and <a href="http://blogs.sun.com/jimmauro">Jim</a>. We were fortunate to have many other subject matter experts also contribute to this book, including <a href="http://blogs.sun.com/mws">Mike Shapiro</a> and <a href="http://blogs.sun.com/jonh">Jon Haslam</a>.<br /><br />For Solaris Performance and Tools, it is exciting to see a book that covers both DTrace and mdb in detail, with practical examples. And Solaris Internals 2nd Edition is fascinating for understanding the kernel, with the numerous Solaris 10 enhancements. I hope people enjoy them both.Brendan Gregghttp://www.blogger.com/profile/03662900777558310512noreply@blogger.com0tag:blogger.com,1999:blog-13965874.post-1153722372098220852006-07-23T23:25:00.000-07:002006-07-23T23:26:12.100-07:00Solaris Internals 2nd Edition - available!Great news! Solaris Internals 2nd Edition is now available to purchase from sites such as <a href="http://www.amazon.com/gp/product/0131482092">Amazon</a>. Amazon writes "Availability: Usually ships within 24 hours" - so in less than a day (I imagine a business day, if you are in the US), you could be holding this landmark book.<br /><br />I've studied this edition from cover to cover, and it is incredible to see such a wide range of topics covered well, and at such a unique depth. It is <i>the</i> reference for the Solaris 10 and OpenSolaris kernel, and is written in an endearing way that makes this enormous topic very approachable.<br /><br />I can't wait to get my hard copy of this book. I really enjoyed studying the content the first time, and I'll really enjoy revisiting the chapters.Brendan Gregghttp://www.blogger.com/profile/03662900777558310512noreply@blogger.com3tag:blogger.com,1999:blog-13965874.post-1153230620317475852006-07-18T05:50:00.000-07:002006-07-18T06:54:43.670-07:00<b>DTrace Slides</b><br /><br />The <a href="http://www.context-switch.com/performance/dtrace_workshop01_slides.pdf">slides</a> I used for the recent DTrace Workshop in London have been put <a href="http://www.context-switch.com/performance/dtrace.htm">online</a> by <a href="http://www.context-switch.com">Context-Switch</a>. This DTrace presentation covers my own method of teaching DTrace, which begins with a summary of performance monitoring tools in Solaris 9 and their limitations. <br /><br />The slides also document many of the strategies I use for DTrace analysis, and is the first time they have been listed together online. They are,<br /><br /><b>DTrace Strategies</b><br /><ol><li>Snoop or Summary</li><br /><li>Drill Down Analysis</li><br /><li>Frequency Count</li><br /><li>Known Count</li><br /><li>Aggregate Stacks</li><br /><li>Elapsed Time</li><br /><li>On-CPU Time</li><br /><li>Milestones</li></ol><br />See chapters 5 and 10 from the slides for details.<br /><br />The slides are used in conjunction with additional live demonstrations and student labs. They also assume that the <i>almost</i> released book, <a href="http://www.solarisinternals.com">Solaris Performance and Tools</a> can be used as a reference.Brendan Gregghttp://www.blogger.com/profile/03662900777558310512noreply@blogger.com0tag:blogger.com,1999:blog-13965874.post-1151463741334950702006-06-27T19:28:00.000-07:002006-06-27T20:02:21.350-07:00<b>Offline</b><br /><br />I've been mostly offline for a few weeks, and this will continue for a few more. I'm in the middle of at 6 weeks of travel (London, London, Perth, Canberra, Melbourne, Tokyo, ...?), much of which is spent at hotels without Internet access or at airports (and all of it without access to my DTrace development servers at home, in Sydney).<br /><br />By a stroke of luck I am actually in Sydney on the 6th of July (inbetween flights), and should be at the next meeting for SOSUG.Brendan Gregghttp://www.blogger.com/profile/03662900777558310512noreply@blogger.com0tag:blogger.com,1999:blog-13965874.post-1149092471926809702006-05-31T09:12:00.001-07:002006-05-31T09:41:11.650-07:00<b>DTraceTazTool - alpha release</b><br /><br />Several years ago, <a href="http://blogs.sun.com/roller/page/rmc">Richard McDougall</a> wrote <a href="http://www.solarisinternals.com/si/tools/taz/index.php">taztool</a> - a GUI to display disk activity in an amazing and intuitive way. It used TNF trace data - a predecessor of <a href="http://www.opensolaris.org/os/community/dtrace">DTrace</a>.<br /><br />In Janurary 2006 I began writing a DTrace version of taztool: DTraceTazTool. I wrote it using GTK and DTrace. Since then, the DTrace JNI has been released, and I've paused developing DTraceTazTool until I've taken a long look at Java 1.5. Until then, I'm releasing where I am at with the GTK/C version, on my DTrace <a href="http://www.brendangregg.com/dtrace.html#DTraceTazTool">website</a>.<br /><br />This is an alpha release - I'm still adding functionality, working through a todo list and fixing bugs. I'm releasing it early as it is already somewhat useful, and a great demo of DTrace. Probably the next feature I'll add is the ability to trace an entire disk at a time (rather than per-slice).<br /><br />The following screenshot shows a UFS file system that was archived using the tar command. For the top plot: the requested block location is on the Y-axis, time is the X-axis, and the colour reflects the number of requests for that location (many == red). The bottom plot shows maximum and average seek distance.<br /><br /><img src="http://users.tpg.com.au/adsln4yb/DTaz/dtaz07.PNG" border=0><br /><br />DTraceTazTool already has some tunable options, such as the size of the pixels drawn and the sample rate,<br /><br /><img src="http://users.tpg.com.au/adsln4yb/DTaz/dtaz06.PNG" border=0><br /><br />DTraceTazTool needs to either run as root, or as a user with the dtrace_kernel privilege.<br /><br />I hope you enjoy the tool so far. I'll post about updates as I complete them.Brendan Gregghttp://www.blogger.com/profile/03662900777558310512noreply@blogger.com0tag:blogger.com,1999:blog-13965874.post-1148837237020155552006-05-28T10:14:00.000-07:002006-05-28T10:27:17.050-07:00<b>Book Updates</b><br /><br />Jim did post an <a href="http://blogs.sun.com/roller/page/jimmauro?entry=a_tale_of_two_books">entry</a> about Solaris Internals 2nd edition and Sun Performance and Tuning. We are all now eagerly awaiting the publication of these books.<br /><br />An article about these books was posted on <a href="http://sunflash.sun.com/articles/99/3">sunflash.sun.com</a> (login required), a website of Sun events.<br /><br />And some updates were made to the Prentice Hall website for these books (<a href="http://vig.prenhall.com/catalog/academic/product/0,1144,0131482092,00.html">Solaris Internals 2nd edition</a>, <a href="http://vig.prenhall.com/catalog/academic/product/0,1144,0131568191,00.html">Sun Performance and Tools</a>).Brendan Gregghttp://www.blogger.com/profile/03662900777558310512noreply@blogger.com0tag:blogger.com,1999:blog-13965874.post-1148831646416687152006-05-28T08:29:00.000-07:002006-05-28T08:54:06.430-07:00<b>DTracing Scheduling Classes</b><br /><br />I've recently been working on some new killer DTrace scripts, and the DTrace <a href="http://www.context-switch.com/performance/dtrace.htm">UK Workshops</a>. I thought I'd take a break and share something interesting.<br /><br />The following is from the <a href="http://www.opensolaris.org/os/community/dtrace/dtracetoolkit">DTraceToolkit</a>, and is one of my favourite tool example files (Docs/Examples/priclass_example.txt). I'm a firm believer in providing examples, and plan to enhance the existing examples in the toolkit further. There also seems to be a trend to include more examples in Solaris man pages (such as zfs and zpool), which is great to see.<br /><br /><hr width=100><br />The following is a demonstration of the <b>priclass.d</b> script.<br /><br />The script was run for several seconds then Ctrl-C was hit. During this time, other processes in different scheduling classes were running.<br /><pre># ./priclass.d<br />Sampling... Hit Ctrl-C to end.<br />^C<br /><br /> IA<br /> value ------------- Distribution ------------- count<br /> 40 | 0<br /> 50 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 30<br /> 60 | 0<br /><br /> SYS<br /> value ------------- Distribution ------------- count<br /> < 0 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 4959<br /> 0 | 0<br /> 10 | 0<br /> 20 | 0<br /> 30 | 0<br /> 40 | 0<br /> 50 | 0<br /> 60 | 30<br /> 70 | 0<br /> 80 | 0<br /> 90 | 0<br /> 100 | 0<br /> 110 | 0<br /> 120 | 0<br /> 130 | 0<br /> 140 | 0<br /> 150 | 0<br /> 160 | 50<br /> >= 170 | 0<br /><br /> RT<br /> value ------------- Distribution ------------- count<br /> 90 | 0<br /> 100 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 110<br /> 110 | 0<br /><br /> TS<br /> value ------------- Distribution ------------- count<br /> < 0 | 0<br /> 0 |@@@@@@@@@@@@@@@ 2880<br /> 10 |@@@@@@@ 1280<br /> 20 |@@@@@ 990<br /> 30 |@@@@@ 920<br /> 40 |@@@@ 670<br /> 50 |@@@@ 730<br /> 60 | 0<br /></pre><br />The output is quite interesting, and illustrates neatly the behaviour of different scheduling classes.<br /><br />The IA interactive class had 30 samples of a 50 to 59 priority, a fairly high priority. This class is used for interactive processes, such as the windowing system. I had clicked on a few windows to create this activity.<br /><br />The SYS system class has had 4959 samples at a < 0 priority - the lowest, which was for the idle thread. There are a few samples at higher priorities, including some in the 160 to 169 range (the highest), which are for interrupt threads. The system class is used by the kernel.<br /><br />The RT real time class had 110 samples in the 100 to 109 priority range. This class is designed for real-time applications, those that must have a consistant response time regardless of other process activity. For that reason, the RT class trumps both TS and IA. I created these events by running "prstat -R" as root, which runs prstat in the real time class.<br /><br />The TS time sharing class is the default scheduling class for the processes on a Solaris system. I ran an infinite shell loop to create heavy activity, "while :; do :; done", which shows a profile that leans towards lower priorities. This is deliberate behaivour from the time sharing class, which reduces the priority of CPU bound processes so that they interefere less with I/O bound processes. The result is more samples in the lower priority ranges.Brendan Gregghttp://www.blogger.com/profile/03662900777558310512noreply@blogger.com0tag:blogger.com,1999:blog-13965874.post-1147155282778731572006-05-08T22:11:00.000-07:002006-05-10T08:04:27.693-07:00<b>Ged - GUI-ed</b><br /><br />I've just uploaded <a href="http://www.brendangregg.com/specials.html#ged">Ged</a> to the Specials collection - it is a GUI front-end to the "ed" line editor, and has been written for Java 1.5 using Swing. It enhances "ed" in several ways, such as providing optional themes, sound effects, and a stunning splash screen rendered in <a href="http://www.povray.org">POV-ray</a>.[1]<br /><br /><img src="http://users.tpg.com.au/adsln4yb/Specials/ged01.png"><br /><br />ok, the real reason I'm coding in Java/Swing is to test drive the new Java 1.5 features and the Swing API. I've coded in Java before, however the last few years I've been using mostly Shell, Perl, C and DTrace. There are some really great features in Java 1.5 to try out. I'm also planning on writing some more freeware GUI-based programs; such as performance monitoring tools (<a href="http://www.opensolaris.org/os/community/dtrace">DTrace</a>) and <a href="http://www.opensolaris.org/os/community/games">Games</a>.<br /><br />In the past I've coded in OSF/Motif, GTK 2.0, Java/AWT and now Java/Swing. It's hard to choose a single winner - they each have different pros and cons; however I do like coding in both GTK and Swing.<br /><br />For now, enjoy Ged. No, I didn't write "ed" itself in Java - it borrows /bin/ed. Of course - I could write "ed" entirerly in Java, which would make Ged more portable, but there are higher things on my todo list!<br /><br />[1] which was then mutilated by downsampling to "classic" 2-bit colour.Brendan Gregghttp://www.blogger.com/profile/03662900777558310512noreply@blogger.com2tag:blogger.com,1999:blog-13965874.post-1145983648237766892006-04-25T09:39:00.000-07:002006-04-25T09:47:31.430-07:00<b>Learn DTrace @ London</b><br /><br />During the past five years I've delivered many training events across Australia and Asia; however this will be my first event in Europe.<br /><br />I will be delivering two 3-day classes near the London area (Egham), from June 7th-9th and June 12th-14th, 2006. The website is <a href="http://www.context-switch.com/performance/dtrace.htm">here</a>, and seats are currently available (but won't be forever).<br /><br />These are special classes that will cover not only DTrace, but most importantly the "DTrace Mentality" - strategies used to tackle problems, and a workshop of mystery problems to practice on. I'll also discuss the DTraceToolkit and other performance tools.<br /><br />There are currently no plans to run these classes again, so as far as I know - this is it.<br /><br />I'm really looking forward to these events. And beforehand, please think of what problems have stumped you in the past on Solaris - especially statistics that you were unable to measure, so that you can ask me in person. If I can't answer it, I'll try to write the script that can. And if the script is useful, it may end up in the DTraceToolkit!Brendan Gregghttp://www.blogger.com/profile/03662900777558310512noreply@blogger.com0tag:blogger.com,1999:blog-13965874.post-1145980673002539802006-04-25T08:05:00.000-07:002006-04-25T08:57:53.036-07:00<b>DTraceToolkit ver 0.96</b><br /><br />I've just uploaded the latest build of <a href="http://www.opensolaris.org/os/community/dtrace/dtracetoolkit">DTraceToolkit</a>, which now contains 104 scripts!<br /><br />12 scripts have been added and 46 scripts were updated - this is a BIG revision. Since there have been so many new changes, I'm holding the version number below 1.00 until the dust has settled (I was keeping version number equal to the script count, however I'll hold back on that for a while).<br /><br />This is the first rev to include a few scripts written by other people - Richard McDougall and Jonathan Adams, who can both write DTrace well and more importantly, have an exceptional understanding of subtle kernel nuances.<br /><br />This rev has had another small style change (which is why so many scripts were updated). I've also began writing a document called the "DTraceToolkit Style Guide", which documents the style and programming best practices used by the scripts in the toolkit. I'll post more about that later.<br /><br />Stefan Parvu helped out with platform testing, which is crucial. Some of the tcp scripts had began not working on newer builds (since they are fbt based, and fbt is not a stable interface (and not supposed to be)). They have been fixed (for now!).<br /><br />As usual, check locations such as Docs/Examples for examples of the newly added scripts in action. Enjoy!Brendan Gregghttp://www.blogger.com/profile/03662900777558310512noreply@blogger.com0tag:blogger.com,1999:blog-13965874.post-1145455286568107402006-04-19T05:36:00.000-07:002006-04-19T07:01:26.620-07:00<b>Tokyo, Japan</b><br /><br />A few weeks ago I was in Tokyo to deliver a couple of Solaris 10 events for Sun. It was the time of year for cherry blossoms, and a great time to take <a href="http://www.brendangregg.com/Photos/Tokyo/">photos</a>. I've just added a series of them to the cityscapes on my website.Brendan Gregghttp://www.blogger.com/profile/03662900777558310512noreply@blogger.com0tag:blogger.com,1999:blog-13965874.post-1144214150446064702006-04-04T20:25:00.000-07:002006-04-04T22:15:50.463-07:00<b>Solaris Internals 2nd Edition</b><br /><br />This book (now 2 volumes) has been sent to the publisher! <br /><br />Jim and Richard have again made a remarkable effort to create a unique and valuable reference. It covers an incredible range of topics, each with useful explinations you won't find anywhere else. <br /><br />Richard has posted an entry to <a href="http://blogs.sun.com/roller/page/rmc?entry=performance_observability_dtrace_and_mdb">describe the books</a>, and links to their tables of contents.<br /><br />I'm very proud to have helped out, including authoring material for the 2nd volume. For many years I've taught Solaris performance classes for Sun, engaged in Solaris performance consulting, and more recently written freeware performance tools - such as the DTraceToolkit. It's been a privilege to share my most useful knowledge in this way.<br /><br />I have provided scripts inline (listing the source) wherever possible - giving the reader the choice of either using the script, or reading and understanding the script internals - and making their own customisations. We do often discuss such tool internals - which is appropriate for Solaris Internals.<br /><br />I've also carefully studied both volumes in detail - and it is truly incredible what they cover. This will open the lid on many new topics that have previously not been discussed or exposed. Other engineers and experts have helped out, providing many valuable insights - especially when discussing technology that they themselves created.<br /><br />I think this has a wider audience than the first edition as we discuss topics closer to home for sysadmins, support staff and developers - such as performance observability and debugging. Check the table of contents for the second volume to see what I mean.<br /><br />It's now up to the publisher to print out the book. If there is any news on this we can share, we will. Check our blogs (<a href="http://blogs.sun.com/rmc">Richard</a>, <a href="http://blogs.sun.com/jimmauro">Jim</a>, and me).<br /><br />I've been somewhat quiet on the OpenSolaris scene during the last several weeks - this is why! I'm now "back" :)Brendan Gregghttp://www.blogger.com/profile/03662900777558310512noreply@blogger.com0tag:blogger.com,1999:blog-13965874.post-1139878239415518132006-02-13T16:24:00.000-08:002006-02-13T16:50:39.426-08:00<b>Learn DTrace, visit Sydney</b><br /><br />Sun is borrowing me to deliver a 3-day <a href="http://au.sun.com/training/catalog/courses/SA-327-S10.xml">DTrace class</a> in Sydney, starting on the 6th of March (assuming enough students sign up). I love DTrace, and teaching it on behalf of Sun is a great privilege.<br /><br />Occasionally, Australian IT workers will be lucky enough to visit the US (or elsewhere) to attend unique training classes. Classes that aren't taught anywhere else in the World. While it's no holiday (it can be hard work), it does make for a fantastic break from sitting behind a desk day after day (or sitting in a cold server room).<br /><br />Please let me return the favour and give you an excuse to visit Sydney, Australia! Where else in the World will you be taught DTrace from the author of the <a href="http://www.opensolaris.org/os/community/dtrace/dtracetoolkit">DTraceToolkit</a>?<br /><br />I have taught DTrace before: be prepared for some really intense training! We usually drink lots of coffee, finish later than usual, and cover a lot more than is in the notes. I encourage students to bring in their work problems and get me to solve them (live, in front of the class!).<br /><br />Ok, I know many companies may choke at the thought of paying for flights to Australia + hotel + training. But if you do wangle it, then I'll see you on the 6th (and the weather is usually great that time of year. ;)Brendan Gregghttp://www.blogger.com/profile/03662900777558310512noreply@blogger.com0