1、SystemTapLinux下的万能观测工具,褚霸核心系统数据库组http:/yufeng.info2010/11/18,Agenda,介绍SystemTap安装和系统要求实践例子参考和杂项结论,SystemTap是什么？,According to http:/sourceware.org/systemtap/SystemTap provides free software (GPL) infrastructure tosimplify the gathering of information about the runningLinux system. This assists diagno
2、sis of a performance orfunctional problem. SystemTap eliminates the need forthe developer to go through the tedious and disruptiveinstrument, recompile, install, and reboot sequence thatmay be otherwise required to collect data.观察活体系统最佳工具，前提是你懂得如何观察！,SystemTap是如何工作的,1. write or choose a script descr
3、ibing what you want to observe2. stap translates it into a kernel module3. stap loads the module and communicates with it4. just wait for your data,五步走,#stap -uv test.stpPass 1: parsed user script and 74 library script(s) using 86868virt/20488res/1792shr kb, in 190usr/20sys/209real ms.Pass 2: analyz
4、ed script: 1 probe(s), 0 function(s), 0 embed(s), 0 global(s) using 87264virt/21148res/1976shr kb, in 10usr/0sys/7real ms.Pass 3: translated to C into /tmp/stapz2iv97/stap_aef621603e006af62084b361e0a0c981_553.c using 87264virt/21332res/2144shr kb, in 0usr/0sys/0real ms.Pass 4: compiled C into stap_a
5、ef621603e006af62084b361e0a0c981_553.ko in 1230usr/160sys/1384real ms.Pass 5: starting run.Pass 5: run completed in 10usr/20sys/12331real ms.,SystemTap 探测点例子,SystemTap is all about executing certain actions when hitting certain probe points.syscall.readwhen entering read() system callsyscall.close.re
6、turnwhen returning from the close() system callmodule(floppy).function(*)when entering any function from the floppy modulekernel.function(*net/socket.c).returnwhen returning from any function in le net/socket.ckernel.statement(*kernel/sched.c:2917)when hitting line 2917 of le kernel/sched.c,更多探测点例子,
7、timer.ms(200) every 200 millisecondsprocess(/bin/ls).function(*) when entering any function in /bin/ls (not its libraries orsyscalls)process(/lib/libc.so.6).function(*malloc*)when entering any glibc function which has malloc in its namekernel.function(*exit*).returnwhen returningfrom any kernel func
8、tion which has exit in its nameRTFM for more (man stapprobes).,SystemTap编程语言,mostly C-style syntax with a feeling of awkbuiltin associative arraysbuiltin aggregates of statistical datavery easy to collect data and do statistics on it (average, min, max, count,. . . )many helper functions (builtin an
9、d in tapsets)RTFM: SystemTap Language Reference shipped with SystemTap(langref.pdf),Performances and safety,language-level safety featuresno pointersno unbounded loopstype inferenceyou can also write probe handlers in C (with -g) but dont complain if you break stuffruntime safety featuresstap enforc
10、es maximum run time for each probe handlervarious concurrency constraints are enforcedoverload processing (dont allow stap to take up all the CPU time)many things can be overriden manually if you really wantsee SAFETY AND SECURITY section of stap(1)The overhead depends a lot of what you are trying t
11、o do but in general stap will try to stop you from doing something stupid (butthen you can still force it to do it).,Some helper functions youll see a lot,pid() which process is this?uid() which user is running this?execname() what is the name of this process?tid() which thread is this?gettimeofday_
12、s() epoch time in secondsprobefunc() what function are we in?print_backtrace() figure out how we ended up hereThere are many many more. RTFM (man stapfuncs) and explore/usr/share/systemtap/tapset/.,Some cool stap options,-x trace only speci ed PID (only for userland probing)-c run given command and
13、only trace it and its children(will still trace all threads for kernel probes)-L list probe points matching given pattern along withavailable variables-d load given module debuginfo to help with symbol resolution in backtraces-g embed C code in stap script unsafe, dangerous and fun,Agenda,介绍SystemTa
14、p安装和系统要求实践例子参考结论,Requirements,SystemTap探测用户空间程序需要utrace的支持，但是这个特性还没有被Linux上游吸收。Redhat的发行版本目前支持这个特性。源码级别跟踪需要安装符号信息包层面需要安装package-debuginfo on RPM distros用户自己的程序需要gcc-g -gdwarf-2 -g3编译stap脚本是编译成内核模块运行的，需要root权限,安装SystemTap,RHEL5U4需要安装内核符号信息:rpm -i kernel-debuginfo-common-2.6.18-164.el5.x86_64.rpmrpm -
15、i kernel-debuginfo-2.6.18-164.el5.x86_64.rpm由于5U4带的SystemTap是0.97版本，需要升级到1.3:./configure prefix=/usr & make & make install如何验证是否成功：# stap topsys.stpSYSCALL COUNT read 48 fcntl 42 . fstat 1-,Agenda,介绍SystemTap安装和系统要求实践例子参考和杂项结论,Example: 谁在执行我们的程序,Listing: exec.stpprobe syscall.exec*printf(exec %s %sn
16、, execname(), argstr)$ stap -L syscall.exec*syscall.execve name:string filename:string args:string argstr:string $filename:char* $argv:char* $envp:char* $regs:struct pt_regs*# stap exec.stpexec sshd /usr/sbin/sshd -Rexec sshd /bin/bash,例子: 谁杀了我的程序,Listing: sigkill.stpprobe signal.sendif(sig_name = S
17、IGKILL)printf(%s was sent to %s (pid:%d) by %s uid :%dn, sig_name, pid_name , sig_pid, execname(), uid()# kill -9 pgrep top#stap sigkill.stp SIGKILL was sent to top (pid:19281) by bash uid :50920,Example tac.c: 工具函数,#include #include #include char* haha = wahahan;char* read_line(FILE* fp, char* buf,
18、 size_t len) return fgets(buf, len, fp);char* reverse_line(char* line, size_t l) char *s = line, *e = s + l - sizeof(n), t; while(s _retvalue =(long) (char*)THIS-argv)1;%probe process(a.out).function(main) filename = stdin; if($argc 1) filename = user_string(get_argv_1($argv); println(filename);,Exa
19、mple 1 continued:,#echo hi|./a.out #./a.out tac.c#stap -gu ./ex1.stp:)stdintac.c,Example 2: callgraph for anything,function trace(entry_p, extra) %( $# 1 %? if (tid() in trace) %) printf(%s%s%s %sn, thread_indent (entry_p), (entry_p0?-:main argc=0x1 argv=0x7fff351ee0c8 30 a.out(18123): -readline fp=
20、0x3f7bb516a0 buf=0x7fff351ecfd0 len=0x1000 590 a.out(18123): reverse_line line=0x7fff351ecfd0 l=0x3 625 a.out(18123): write_line line=0x7fff351ecfd0 731 a.out(18123): readline fp=0x3f7bb516a0 buf=0x7fff351ecfd0 len=0x1000 762 a.out(18123): -readline return=0x0 770 a.out(18123):-main return=0x0,Examp
21、le 3:获取行长度,global line_lenprobe process(a.out).statement(reverse_linetac.c+1) line_len 0) print(hist_linear(line_len, 8, 128, 8);,Example 3 continued:,# ls -al|./a.out#./ex3.stp:)value |- count line, abcdefgn);%probe process(a.out).function(reverse_line)line = user_string($line);probe process(a.out)
22、.function(reverse_line).return if(isinstr(line, tac) $return = $haha;else if (isinstr(line, hello) alert_line($return);,Example 6 continued:,# stap ./ex6.stp#echo tac|./a.out wahaha#echo hello|./a.out abcdefg#echo world|./a.out dlrow,Agenda,介绍SystemTap安装和系统要求实践例子参考和杂项结论,Emacs Systemtap mode,在这里下载 sy
23、stemtap-mode.el: http:/coderepos.org/share/browser/lang/elisp/systemtap-mode/systemtap-mode.el?format=txt 在.emacs里面添加以下二行： (autoload systemtap-mode systemtap-mode) (add-to-list auto-mode-alist (.stp$ . systemtap-mode),参考文献,http:/sourceware.org/systemtap/langref/http:/sourceware.org/systemtap/tapsets/http:/ is often described as DTrace for Linux.OProfile takes sample every $N CPU cycles so you can try to fi gureout what each CPU is spending its time on.SystemTap ，居家必备！,谢谢大家！,Any question？,
