【997755.com澳门葡京】Tookit工具包之pt

 

 

 

 

Preface

Preface

Preface

Preface

 

 

 

 

    We’ve got a lot of files related with
system performance which generated by pt-stalk in directory
“/var/lib/pt-stalk” in my last blog.We can use “cat”,”more”,”less” to
print the contents of them but it is really not convenient to do
so.

    We have a lot of methods to diagnose
problems in our system such as strace,pstack,gstack,gdb,pt-pmp,etc.But
sometimes there will be some fitful performance issues which are not so
easy to trace.Thus,pt-stalk may help us in diagnosing these kind of
problems.

    Sometimes we need to know the details
of a program(porcess) when it is running even it’s on the edge of
crach.There’s a lot of tools can be used to do these kind of job such as
truss,strace,ltrace,pstack,gstack,gdb,etc.Truss and strace is focus on
system call and signal generation whereas pstack,gstack,gdb are relevant
with function call.Actually,pstack is a soft link of gstack,and gstack
is a shell packaging script of gdb.

    We are always obliged to analyze many
outputs generated by various tools directly even in flat text files in
server performance diagnosis.Further more,we need to compare the
differences depend on these results such as the output of “mysqladmin”
tool.

 

 

 

 

Introduce

Introduce

Introduce

Introduce

 

 

 

 

    pt-sift is the right tool helping us to
retrieve diagnostic data easily.There’re poor options of the tool to
learn.Therefor,it’s easy enough to sift the contents we need with no
time.**

    pt-stalk is a tool to collect detail
diagnostic data base on triggers you specified such as
gdb,oprofile,strace,tcpdump.The trigger is not the conception of trigger
in database.They’re different at all.pt-stalk provides various options
to collect comprehensive data you need.It’s really useful and
helpful in performance diagnosing.Let’s see some details of it.

    pt-pmp is also relies on linux
gdb.It’s a tool which aggregate gdb statck traces for a specified
program especially for MySQL.pt-pmp can create and summarize full stack
traces of linux process by which we can diagnose what the process is
waiting for.It is mostly used in performance tuning
scenarios.

    pt-mext is tht very tool which can
help you to accomplish the work mentioned above 
easily.It can execute the command
which you specify then print the result side-by-side in one line.Not
only can it print results from a running program but also can it print
results from a flat text file.

 

 

 

 

Procedure

Procedure

Procedure

Procedure

 

 

 

 

Usage

Usage

Usage

Usage

1 pt-sift FILE|PREFIX|DIRECTORY
1 pt-stalk [OPTIONS]
1 pt-pmp [OPTIONS] [FILES]
1 pt-mext [OPTIONS] -- COMMAND

 

 

 

 

Examples

**Common Parameters**

**Parameters(few parameters in this
tool)【997755.com澳门葡京】Tookit工具包之pt。**

**Common Parameters**

 

 1 --collect //Collect diagostic data based on spedified triggers.
 2 --collect-gdb //Collect gdb data.It prints stack traces from all threads.
 3 --collect-oprofile //Collect oprofile data.
 4 --collect-strace //Collect strace data.Do not specify it when "--collect-gdb" has been used.
 5 --collect-tcpdump //Collect tcpdump data.
 6 --cycles //Limite the times when triggering condition occurs(default "5").
 7 --dest //Specify the position to store diagnostic data(default "/var/lib/pt-stalk").
 8 --disk-bytes-free //Prevent the lack of least disk space.It does not collect data until it has the necessary disk space(default "100M",valid surffixes of unit is k,M,G and T).
 9 --disk-pct-free //It's similar with "--disk-bytes-free" but specify the percentage of disk space.
10 --function //Specify the contents to watch for the triggers(default "status",other value is "processlist" or <yourfilename>).
11 --iterations //Limit the times to collect.It will run forever if not give a specific value.
12 --log //Specify the postion to record logs(default "/var/log/pt-stalk.log").It only generates when deamonized.
13 --match //The pattern to specify when using "--function processlist".
14 --mysql-only //Merely collect the MySQL relevant diagnostic data but disk space is the exception.
15 --retention-time //Specify the purge days of diagnostic data(default "30").
16 --run-time //Specify how many seconds to collect diagnostic data(default "30").It should not be longer than the value of "--sleep".
17 --sleep //Specify how many seconds to sleep after last collection.It used to prevent too many operations of collection(default "300").
18 --stalk //Watch and wait for trigger occur(default "yes").
19 --no_stalk //The option can be specify if you want to collect diagnostic data immediately without waiting for trigger to occur.
20 --threshold //Specify the maximum value for collecting(default "25").
21 --variable //Specify the variables to compared against(default "Threads_running").
1 --binary //Specify which binary program to trace(default 'mysqld').
2 --pid //Specify a process id to trace.
3 --iterations //Specify trace times(default "1").
4 --interval //Number of seconds every time(default "0"). 
5 --lines //Specify the number of functions to trace(default "0" means infinite).
6 --save-samples //output aggregation into a file for later analysis.
1 --relative //Each column substracts the value of the column before it.

Print the output of the files
generated by pt-stalk.

 

 

 

 1 [root@zlm2 09:50:42 /var/lib/pt-stalk]
 2 #pt-sift
 3 Error:  is not a directory, and there are no pt-stalk files in the curent working directory (/var/lib/pt-stalk) with a  prefix.
 4 For more information, 'man pt-sift' or 'perldoc /usr/bin/pt-sift'.
 5 
 6 [root@zlm2 09:50:44 /var/lib/pt-stalk]
 7 #pt-sift /var/lib/pt-stalk
 8 
 9   2018_07_09_08_19_39  2018_07_09_08_24_39
10 
11 Select a timestamp from the list [2018_07_09_08_24_39] 2018_07_09_08_19_39
12 ======== zlm2 at 2018_07_09_08_19_39 DEFAULT (1 of 2) ========
13 --diskstats--
14   #ts device    rd_s rd_avkb rd_mb_s rd_mrg rd_cnc   rd_rt    wr_s wr_avkb wr_mb_s wr_mrg wr_cnc   wr_rt busy in_prg    io_s  qtime stime
15  {29} sda        2.4    16.9     0.0     0%    0.0     2.6    26.2    48.9     1.2    10%    0.1     4.7   3%      0    28.6    3.5   1.1
16  sda  0% 5% 0% . . . . . . . . . . . . . . . . . . . . . 20% 0% . . . .
17 --vmstat--
18 r b swpd  free buff  cache si so  bi   bo   in    cs us sy id wa st
19 7 5 5276 73004    0 569412  0  0  10  117   74   433  2  1 97  0  0
20 4 0 5604 72876    0 571420  2 11 213 1261 1423 11833 80 20  0  0  0
21 wa 0% . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
22 --innodb--
23     txns: 3xnot (0s)
24     0 queries inside InnoDB, 0 queries in queue
25     Main thread: sleeping, pending reads 0, writes 0, flush 0
26     Log: lsn = 1144708398, chkp = 1127336399, chkp age = 17371999
27     Threads are waiting at:
28     Threads are waiting on:
29 --processlist--
30     State
31       6  
32       1  starting
33       1  executing
34     Command
35       6  Sleep
36       2  Query
37 --stack traces--
38     No stack trace file exists
39 --oprofile--
40     No opreport file exists
41     
42 //It won't exit until you input "q".
43 //Further more,you can input other keystrokes to navigate the contents you want.

Examples

Exapmle

Example

 

 

 

 

**Check the disk relevant output by press
“q” in above output.**

Generate benchmark with
sysbench.

Trace gdb stack of mysqld into a
file(start mysqld and then execute pt-pmp).

Collect the global status of MySQL
with 1s interval and 3 times in all by mysqladmin.

 1 ======== zlm2 at 2018_07_09_08_19_39 DISKSTATS (1 of 2) ========
 2 Starting /usr/bin/pt-diskstats
 3 
 4   #ts device    rd_s rd_avkb rd_mb_s rd_mrg rd_cnc   rd_rt    wr_s wr_avkb wr_mb_s wr_mrg wr_cnc   wr_rt busy in_prg    io_s  qtime stime
 5   1.0 sda       41.2    18.2     0.7     0%    0.1     3.4    24.7    35.0     0.8     0%    0.0     1.8   6%      0    65.8    1.9   0.9
 6   1.0 sda2      41.2    18.2     0.7     0%    0.1     3.4    20.6    42.0     0.8     0%    0.0     1.1   4%      0    61.7    2.0   0.6
 7   1.0 dm-0      41.2    18.2     0.7     0%    0.1     3.4    20.6    42.0     0.8     0%    0.0     2.1   6%      0    61.7    2.0   0.9
 8 
 9 //too many lines here,they've been ommited.
10 
11   #ts device    rd_s rd_avkb rd_mb_s rd_mrg rd_cnc   rd_rt    wr_s wr_avkb wr_mb_s wr_mrg wr_cnc   wr_rt busy in_prg    io_s  qtime stime
12  28.0 sda        0.0     0.0     0.0     0%    0.0     0.0    18.4    29.9     0.5     0%    0.0     1.0   2%      0    18.4    0.1   0.9
13  28.0 sda2       0.0     0.0     0.0     0%    0.0     0.0    14.6    37.9     0.5     0%    0.0     0.6   1%      0    14.6    0.1   0.5
14  28.0 dm-0       0.0     0.0     0.0     0%    0.0     0.0    14.6    37.9     0.5     0%    0.0     1.3   2%      0    14.6    0.1   1.1
15  28.0 dm-1       0.0     0.0     0.0     0%    0.0     0.0     0.0     0.0     0.0     0%    0.0     0.0   0%      0     0.0    0.0   0.0
16 
17   #ts device    rd_s rd_avkb rd_mb_s rd_mrg rd_cnc   rd_rt    wr_s wr_avkb wr_mb_s wr_mrg wr_cnc   wr_rt busy in_prg    io_s  qtime stimePress a key to continue or choose a different action
 1 [root@zlm2 08:07:54 ~/sysbench-1.0/src/lua]
 2 #sysbench oltp_read_write.lua --mysql-host=192.168.1.101 --mysql-port=3306 --mysql-user=zlm --mysql-password=aaron8219 --mysql-db=sysbench --tables=10 --table-size=100000 --mysql-storage-engine=innodb cleanup
 3 sysbench 1.0.15 (using bundled LuaJIT 2.1.0-beta2)
 4 
 5 Dropping table 'sbtest1'...
 6 Dropping table 'sbtest2'...
 7 Dropping table 'sbtest3'...
 8 Dropping table 'sbtest4'...
 9 Dropping table 'sbtest5'...
10 Dropping table 'sbtest6'...
11 Dropping table 'sbtest7'...
12 Dropping table 'sbtest8'...
13 Dropping table 'sbtest9'...
14 Dropping table 'sbtest10'...
15 
16 [root@zlm2 08:08:08 ~/sysbench-1.0/src/lua]
17 #sysbench oltp_read_write.lua --mysql-host=192.168.1.101 --mysql-port=3306 --mysql-user=zlm --mysql-password=aaron8219 --mysql-db=sysbench --tables=10 --table-size=100000 --mysql-storage-engine=innodb prepare
18 sysbench 1.0.15 (using bundled LuaJIT 2.1.0-beta2)
19 
20 Creating table 'sbtest1'...
21 Inserting 100000 records into 'sbtest1'
22 Creating a secondary index on 'sbtest1'...
23 Creating table 'sbtest2'...
24 Inserting 100000 records into 'sbtest2'
25 Creating a secondary index on 'sbtest2'...
26 Creating table 'sbtest3'...
27 Inserting 100000 records into 'sbtest3'
28 Creating a secondary index on 'sbtest3'...
29 Creating table 'sbtest4'...
30 Inserting 100000 records into 'sbtest4'
31 Creating a secondary index on 'sbtest4'...
32 Creating table 'sbtest5'...
33 Inserting 100000 records into 'sbtest5'
34 Creating a secondary index on 'sbtest5'...
35 Creating table 'sbtest6'...
36 Inserting 100000 records into 'sbtest6'
37 Creating a secondary index on 'sbtest6'...
38 Creating table 'sbtest7'...
39 Inserting 100000 records into 'sbtest7'
40 Creating a secondary index on 'sbtest7'...
41 Creating table 'sbtest8'...
42 Inserting 100000 records into 'sbtest8'
43 Creating a secondary index on 'sbtest8'...
44 Creating table 'sbtest9'...
45 Inserting 100000 records into 'sbtest9'
46 Creating a secondary index on 'sbtest9'...
47 Creating table 'sbtest10'...
48 Inserting 100000 records into 'sbtest10'
49 Creating a secondary index on 'sbtest10'...
50 
51 [root@zlm2 08:09:23 ~/sysbench-1.0/src/lua]
52 #sysbench oltp_read_write.lua --mysql-host=192.168.1.101 --mysql-port=3306 --mysql-user=zlm --mysql-password=aaron8219 --mysql-db=sysbench --threads=3 --time=3600 --report-interval=10 --rand-type=uniform run
53 sysbench 1.0.15 (using bundled LuaJIT 2.1.0-beta2)
54 
55 Running the test with following options:
56 Number of threads: 3
57 Report intermediate results every 10 second(s)
58 Initializing random number generator from current time
59 
60 
61 Initializing worker threads...
62 
63 Threads started!
64 
65 [ 10s ] thds: 3 tps: 258.68 qps: 5176.49 (r/w/o: 3624.11/1034.82/517.56) lat (ms,95%): 15.83 err/s: 0.00 reconn/s: 0.00
66 [ 20s ] thds: 3 tps: 286.25 qps: 5726.15 (r/w/o: 4008.67/1144.89/572.60) lat (ms,95%): 15.00 err/s: 0.00 reconn/s: 0.00
67 [ 30s ] thds: 3 tps: 270.82 qps: 5416.13 (r/w/o: 3790.80/1083.69/541.64) lat (ms,95%): 16.12 err/s: 0.00 reconn/s: 0.00
68 [ 40s ] thds: 3 tps: 280.98 qps: 5619.75 (r/w/o: 3934.26/1123.53/561.97) lat (ms,95%): 16.12 err/s: 0.00 reconn/s: 0.00
69 [ 50s ] thds: 3 tps: 298.42 qps: 5968.38 (r/w/o: 4177.83/1193.70/596.85) lat (ms,95%): 14.46 err/s: 0.00 reconn/s: 0.00
70 [ 60s ] thds: 3 tps: 278.88 qps: 5578.20 (r/w/o: 3904.92/1115.52/557.76) lat (ms,95%): 15.83 err/s: 0.00 reconn/s: 0.00
71 [ 70s ] thds: 3 tps: 280.91 qps: 5617.66 (r/w/o: 3932.21/1123.63/561.82) lat (ms,95%): 15.83 err/s: 0.00 reconn/s: 0.00
72 [ 80s ] thds: 3 tps: 281.68 qps: 5632.83 (r/w/o: 3942.77/1126.71/563.35) lat (ms,95%): 16.12 err/s: 0.00 reconn/s: 0.00
73 [ 90s ] thds: 3 tps: 281.60 qps: 5631.55 (r/w/o: 3942.07/1126.39/563.10) lat (ms,95%): 16.12 err/s: 0.00 reconn/s: 0.00
74 [ 100s ] thds: 3 tps: 287.62 qps: 5753.86 (r/w/o: 4028.02/1150.49/575.35) lat (ms,95%): 15.55 err/s: 0.00 reconn/s: 0.00
75 [ 110s ] thds: 3 tps: 308.99 qps: 6180.45 (r/w/o: 4326.12/1236.35/617.97) lat (ms,95%): 13.95 err/s: 0.00 reconn/s: 0.00
76 ... //Omitted.
 1 [root@zlm1 03:01:30 ~]
 2 #pt-pmp --save-samples pmp.log
 3 Sun Jul  1 03:21:06 CEST 2018
 4 /usr/bin/pt-pmp: line 665: gdb: command not found //gdb package is not installed.
 5       1 
 6 
 7 [root@zlm1 03:21:06 ~]
 8 #yum install gdb //Install gdb package first.
 9 //Omitted
10 Installed:
11   gdb.x86_64 0:7.6.1-110.el7                                                                                                                         
12 
13 Complete!
14 
15 [root@zlm1 03:23:19 ~]
16 #pt-pmp --save-samples pmp.log
17 Sun Jul  1 03:23:22 CEST 2018
18      10 __io_getevents_0_4(libaio.so.1),LinuxAIOHandler::collect(os0file.cc:2500),LinuxAIOHandler::poll(os0file.cc:2646),os_aio_linux_handler(os0file.cc:2702),os_aio_handler(os0file.cc:2702),fil_aio_wait(fil0fil.cc:5835),io_handler_thread(srv0start.cc:311),start_thread(libpthread.so.0),clone(libc.so.6)
19       3 pthread_cond_wait,wait(os0event.cc:165),os_event::wait_low(os0event.cc:165),srv_worker_thread(srv0srv.cc:2520),start_thread(libpthread.so.0),clone(libc.so.6)
20       1 sigwait(libpthread.so.0),signal_hand(mysqld.cc:2120),pfs_spawn_thread(pfs.cc:2190),start_thread(libpthread.so.0),clone(libc.so.6)
21       1 sigwaitinfo(libc.so.6),timer_notify_thread_func(posix_timers.c:77),pfs_spawn_thread(pfs.cc:2190),start_thread(libpthread.so.0),clone(libc.so.6)
22       1 pthread_cond_wait,wait(os0event.cc:165),os_event::wait_low(os0event.cc:165),srv_purge_coordinator_suspend(srv0srv.cc:2676),srv_purge_coordinator_thread(srv0srv.cc:2676),start_thread(libpthread.so.0),clone(libc.so.6)
23       1 pthread_cond_wait,wait(os0event.cc:165),os_event::wait_low(os0event.cc:165),buf_resize_thread(buf0buf.cc:3016),start_thread(libpthread.so.0),clone(libc.so.6)
24       1 pthread_cond_wait,wait(os0event.cc:165),os_event::wait_low(os0event.cc:165),buf_dump_thread(buf0dump.cc:782),start_thread(libpthread.so.0),clone(libc.so.6)
25       1 pthread_cond_wait,native_cond_wait(thr_cond.h:140),my_cond_wait(thr_cond.h:140),inline_mysql_cond_wait(thr_cond.h:140),compress_gtid_table(thr_cond.h:140),pfs_spawn_thread(pfs.cc:2190),start_thread(libpthread.so.0),clone(libc.so.6)
26       1 pthread_cond_timedwait,os_event::timed_wait(os0event.cc:285),os_event::wait_time_low(os0event.cc:412),srv_monitor_thread(srv0srv.cc:1585),start_thread(libpthread.so.0),clone(libc.so.6)
27       1 pthread_cond_timedwait,os_event::timed_wait(os0event.cc:285),os_event::wait_time_low(os0event.cc:412),srv_error_monitor_thread(srv0srv.cc:1751),start_thread(libpthread.so.0),clone(libc.so.6)
28       1 pthread_cond_timedwait,os_event::timed_wait(os0event.cc:285),os_event::wait_time_low(os0event.cc:412),pc_sleep_if_needed(buf0flu.cc:2690),buf_flush_page_cleaner_coordinator(buf0flu.cc:2690),start_thread(libpthread.so.0),clone(libc.so.6)
29       1 pthread_cond_timedwait,os_event::timed_wait(os0event.cc:285),os_event::wait_time_low(os0event.cc:412),lock_wait_timeout_thread(lock0wait.cc:501),start_thread(libpthread.so.0),clone(libc.so.6)
30       1 pthread_cond_timedwait,os_event::timed_wait(os0event.cc:285),os_event::wait_time_low(os0event.cc:412),ib_wqueue_timedwait(ut0wqueue.cc:160),fts_optimize_thread(fts0opt.cc:3031),start_thread(libpthread.so.0),clone(libc.so.6)
31       1 pthread_cond_timedwait,os_event::timed_wait(os0event.cc:285),os_event::wait_time_low(os0event.cc:412),dict_stats_thread(dict0stats_bg.cc:428),start_thread(libpthread.so.0),clone(libc.so.6)
32       1 poll(libc.so.6),vio_io_wait(viosocket.c:786),vio_socket_io_wait(viosocket.c:77),vio_read(viosocket.c:132),net_read_raw_loop(net_serv.cc:672),net_read_packet_header(net_serv.cc:756),net_read_packet(net_serv.cc:756),my_net_read(net_serv.cc:899),Protocol_classic::read_packet(protocol_classic.cc:808),Protocol_classic::get_command(protocol_classic.cc:965),do_command(sql_parse.cc:938),handle_connection(connection_handler_per_thread.cc:300),pfs_spawn_thread(pfs.cc:2190),start_thread(libpthread.so.0),clone(libc.so.6)
33       1 poll(libc.so.6),Mysqld_socket_listener::listen_for_connection_event(socket_connection.cc:852),connection_event_loop(connection_acceptor.h:66),mysqld_main(connection_acceptor.h:66),__libc_start_main(libc.so.6),_start
34       1 nanosleep(libpthread.so.0),os_thread_sleep(os0thread.cc:279),srv_master_sleep(srv0srv.cc:2330),srv_master_thread(srv0srv.cc:2330),start_thread(libpthread.so.0),clone(libc.so.6)
35       
36 //Above is the messages outupted on screen.All the contents is different functions relevant with linux and mysqld.
37 //The output sorts with rules of most-frequen first.
38 //The combination of function stacks lead by "__io_getevents_0_4(libaio.so.1)"(which is relvent with io) appears 10 times.
 1 [root@zlm2 10:43:03 ~]
 2 #mysqladmin ext -i1 -c3 > mysqladmin.log
 3 
 4 [root@zlm2 10:43:16 ~]
 5 #ls -l
 6 total 2554836
 7 -rw-------.  1 root root       1431 Jul 16  2015 anaconda-ks.cfg
 8 -rwxr-xr-x   1 root root 2611024896 Jun 13 03:03 mysql-5.7.21-linux-glibc2.12-x86_64.tar
 9 -rw-r--r--   1 root root     108476 Jul 12 10:43 mysqladmin.log
10 -rwxr-xr-x   1 root root         54 Jun 13 04:16 mysqld.sh
11 -rwxr-xr-x   1 root root    5000812 Jun 26 08:59 percona-toolkit-3.0.4-1.el7.x86_64.rpm
12 drwxr-xr-x  13 root root       4096 Jul  4 03:21 sysbench-1.0

 

 

 

 

**Check the innodb status relevant output
by press “q”,then press “i” in above output.**

**Collect diagnostic data using
pt-stalk.**

**Check the content of output
file.**

**The
“mysqladmin.log” file contains all the output of the three times
individually.**

 1 Press a key to continue or choose a different action
 2 ======== zlm2 at 2018_07_09_08_19_39 INNODB (1 of 2) ========
 3 Viewing InnoDB files
 4 ... //Several otmitted blank rows here.
 5 *************************** 1. row ***************************
 6   Type: InnoDB
 7   Name: 
 8 Status: 
 9 =====================================
10 2018-07-09 08:19:40 0x7f7ce8211700 INNODB MONITOR OUTPUT
11 =====================================
12 Per second averages calculated from the last 20 seconds
13 -----------------
14 BACKGROUND THREAD
15 -----------------
16 srv_master_thread loops: 642 srv_active, 0 srv_shutdown, 18648 srv_idle
17 srv_master_thread log flush and writes: 19290
18 ----------
19 SEMAPHORES
20 ----------
21 OS WAIT ARRAY INFO: reservation count 2620
22 OS WAIT ARRAY INFO: signal count 2117
23 RW-shared spins 0, rounds 1009, OS waits 422
24 RW-excl spins 0, rounds 29932, OS waits 1071
25 RW-sx spins 198, rounds 5940, OS waits 137
26 Spin rounds per wait: 1009.00 RW-shared, 29932.00 RW-excl, 30.00 RW-sx
27 ------------
28 TRANSACTIONS
29 ------------
30 Trx id counter 363859
31 Purge done for trx's n:o < 363858 undo n:o < 0 state: running
32 History list length 360
33 LIST OF TRANSACTIONS FOR EACH SESSION:
34 ---TRANSACTION 421649654532384, not started
35 0 lock struct(s), heap size 1136, 0 row lock(s)
36 ... //Omitted.
 1 [root@zlm2 08:14:36 /data/mysql/mysql3306]
 2 #pt-stalk --host localhost --port 3306 --user root --password Passw0rd --collect-gdb --cycles 1 --variable Threads_connect --threshold 2
 3 mysql: [Warning] Using a password on the command line interface can be insecure.
 4 2018_07_09_08_15_46 Starting /usr/bin/pt-stalk --function=status --variable=Threads_connect --threshold=2 --match= --cycles=1 --interval=1 --iterations= --run-time=30 --sleep=300 --dest=/var/lib/pt-stalk --prefix= --notify-by-email= --log=/var/log/pt-stalk.log --pid=/var/run/pt-stalk.pid --plugin=
 5 mysqladmin: [Warning] Using a password on the command line interface can be insecure.
 6 2018_07_09_08_15_46 Detected value is empty; something failed?  Trigger exit status: 0
 7 mysqladmin: [Warning] Using a password on the command line interface can be insecure.
 8 2018_07_09_08_15_47 Detected value is empty; something failed?  Trigger exit status: 0
 9 mysqladmin: [Warning] Using a password on the command line interface can be insecure.
10 2018_07_09_08_15_48 Detected value is empty; something failed?  Trigger exit status: 0
11 mysqladmin: [Warning] Using a password on the command line interface can be insecure.
12 2018_07_09_08_15_50 Detected value is empty; something failed?  Trigger exit status: 0
13 mysqladmin: [Warning] Using a password on the command line interface can be insecure.
14 2018_07_09_08_15_51 Detected value is empty; something failed?  Trigger exit status: 0
15 mysqladmin: [Warning] Using a password on the command line interface can be insecure.
16 2018_07_09_08_15_52 Detected value is empty; something failed?  Trigger exit status: 0
17 mysqladmin: [Warning] Using a password on the command line interface can be insecure.
18 2018_07_09_08_15_53 Detected value is empty; something failed?  Trigger exit status: 0
19 mysqladmin: [Warning] Using a password on the command line interface can be insecure.
20 2018_07_09_08_15_54 Detected value is empty; something failed?  Trigger exit status: 0
21 mysqladmin: [Warning] Using a password on the command line interface can be insecure.
22 2018_07_09_08_15_55 Detected value is empty; something failed?  Trigger exit status: 0
23 mysqladmin: [Warning] Using a password on the command line interface can be insecure.
24 2018_07_09_08_15_56 Detected value is empty; something failed?  Trigger exit status: 0
25 ^C2018_07_09_08_15_57 Caught signal, exiting
26 2018_07_09_08_15_57 All subprocesses have finished
27 2018_07_09_08_15_57 Exiting because OKTORUN is false
28 2018_07_09_08_15_57 /usr/bin/pt-stalk exit status 1
29 
30 //The value of "--variable" should be "Threads_connected".
31 //Modify the correct value and run it again.
32 
33 [root@zlm2 08:15:57 /data/mysql/mysql3306]
34 #pt-stalk --host localhost --port 3306 --user root --password Passw0rd --collect-gdb --cycles 1 --variable Threads_connected --threshold 2
35 mysql: [Warning] Using a password on the command line interface can be insecure.
36 2018_07_09_08_19_39 Starting /usr/bin/pt-stalk --function=status --variable=Threads_connected --threshold=2 --match= --cycles=1 --interval=1 --iterations= --run-time=30 --sleep=300 --dest=/var/lib/pt-stalk --prefix= --notify-by-email= --log=/var/log/pt-stalk.log --pid=/var/run/pt-stalk.pid --plugin=
37 mysqladmin: [Warning] Using a password on the command line interface can be insecure.
38 2018_07_09_08_19_39 Check results: status(Threads_connected)=5, matched=yes, cycles_true=1
39 2018_07_09_08_19_39 Collect 1 triggered
40 2018_07_09_08_19_39 Collect 1 PID 5921
41 2018_07_09_08_19_39 Collect 1 done
42 2018_07_09_08_19_39 Sleeping 300 seconds after collect
43 mysqladmin: [Warning] Using a password on the command line interface can be insecure.
44 2018_07_09_08_24_39 Check results: status(Threads_connected)=6, matched=yes, cycles_true=1
45 2018_07_09_08_24_39 Collect 2 triggered
46 2018_07_09_08_24_39 Collect 2 PID 7398
47 2018_07_09_08_24_39 Collect 2 done
48 2018_07_09_08_24_40 Sleeping 300 seconds after collect
49 ^C2018_07_09_08_25_56 Caught signal, exiting //Terminate to collect diagnostic data by "Ctrl+C"
50 2018_07_09_08_25_56 Waiting up to 90 seconds for subprocesses to finish...
51 2018_07_09_08_25_56 Exiting because OKTORUN is false
52 2018_07_09_08_25_56 /usr/bin/pt-stalk exit status 1
#more pmp.log
TS 845703637.1530408066 2018-07-01 03:21:06
[New LWP 3981]
[New LWP 3875]
[New LWP 3874]
[New LWP 3846]
[New LWP 3845]
[New LWP 3844]
[New LWP 3843]
[New LWP 3842]
[New LWP 3841]
[New LWP 3840]
[New LWP 3839]
[New LWP 3838]
[New LWP 3837]
[New LWP 3836]
[New LWP 3835]
[New LWP 3833]
[New LWP 3832]
[New LWP 3831]
[New LWP 3830]
[New LWP 3829]
[New LWP 3828]
[New LWP 3827]
[New LWP 3826]
[New LWP 3825]
[New LWP 3824]
[New LWP 3823]
[New LWP 3822] //We got 28 new LWP from 3822 to 3981.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
0x00007f50efe74b7d in poll () from /lib64/libc.so.6

Thread 28 (Thread 0x7f50e7ec1700 (LWP 3822)): //It's the last thread in pmp.log file here.
#0  0x00007f50efdbf7da in sigwaitinfo () from /lib64/libc.so.6
#1  0x0000000000f5171b in timer_notify_thread_func (arg=<optimized out>) at /export/home/pb2/build/sb_0-26514852-1514433850.9/mysql-5.7.21/mysys/posix_timers.c:77
#2  0x0000000000f707b4 in pfs_spawn_thread (arg=0x30cb000) at /export/home/pb2/build/sb_0-26514852-1514433850.9/mysql-5.7.21/storage/perfschema/pfs.cc:2190
#3  0x00007f50f13c1df5 in start_thread () from /lib64/libpthread.so.0
#4  0x00007f50efe7f1ad in clone () from /lib64/libc.so.6

Thread 27 (Thread 0x7f50ded07700 (LWP 3823)):
#0  0x00007f50f11b8644 in __io_getevents_0_4 () from /lib64/libaio.so.1
#1  0x000000000106a514 in LinuxAIOHandler::collect (this=0x7f50ded06e10) at /export/home/pb2/build/sb_0-26514852-1514433850.9/mysql-5.7.21/storage/innobase/os/os0file.cc:2500
#2  0x000000000106bc34 in LinuxAIOHandler::poll (this=0x7f50ded06e10, m1=0x7f50ded06ec8, m2=0x7f50ded06ec0, request=0x7f50ded06e90) at /export/home/pb2/build/sb_0-26514852-1514433850.9/mysql-5.7.21/storage/innobase/os/os0file.cc:2646
#3  0x000000000106d65c in os_aio_linux_handler (request=0x7f50ded06e90, m2=0x7f50ded06ec0, m1=0x7f50ded06ec8, global_segment=0) at /export/home/pb2/build/sb_0-26514852-1514433850.9/mysql-5.7.21/storage/innobase/os/os0file.cc:2702
#4  os_aio_handler (segment=0, m1=0x7f50ded06ec8, m2=0x7f50ded06ec0, request=0x7f50ded06e90) at /export/home/pb2/build/sb_0-26514852-1514433850.9/mysql-5.7.21/storage/innobase/os/os0file.cc:6259
#5  0x000000000120196d in fil_aio_wait (segment=0) at /export/home/pb2/build/sb_0-26514852-1514433850.9/mysql-5.7.21/storage/innobase/fil/fil0fil.cc:5835
#6  0x000000000110e980 in io_handler_thread (arg=<optimized out>) at /export/home/pb2/build/sb_0-26514852-1514433850.9/mysql-5.7.21/storage/innobase/srv/srv0start.cc:311
#7  0x00007f50f13c1df5 in start_thread () from /lib64/libpthread.so.0
#8  0x00007f50efe7f1ad in clone () from /lib64/libc.so.6

Thread 26 (Thread 0x7f50de506700 (LWP 3824)):
#0  0x00007f50f11b8644 in __io_getevents_0_4 () from /lib64/libaio.so.1
#1  0x000000000106a514 in LinuxAIOHandler::collect (this=0x7f50de505e10) at /export/home/pb2/build/sb_0-26514852-1514433850.9/mysql-5.7.21/storage/innobase/os/os0file.cc:2500
#2  0x000000000106bc34 in LinuxAIOHandler::poll (this=0x7f50de505e10, m1=0x7f50de505ec8, m2=0x7f50de505ec0, request=0x7f50de505e90) at /export/home/pb2/build/sb_0-26514852-1514433850.9/mysql-5.7.21/storage/innobase/os/os0file.cc:2646
#3  0x000000000106d65c in os_aio_linux_handler (request=0x7f50de505e90, m2=0x7f50de505ec0, m1=0x7f50de505ec8, global_segment=1) at /export/home/pb2/build/sb_0-26514852-1514433850.9/mysql-5.7.21/storage/innobase/os/os0file.cc:2702
#4  os_aio_handler (segment=1, m1=0x7f50de505ec8, m2=0x7f50de505ec0, request=0x7f50de505e90) at /export/home/pb2/build/sb_0-26514852-1514433850.9/mysql-5.7.21/storage/innobase/os/os0file.cc:6259
#5  0x000000000120196d in fil_aio_wait (segment=1) at /export/home/pb2/build/sb_0-26514852-1514433850.9/mysql-5.7.21/storage/innobase/fil/fil0fil.cc:5835
#6  0x000000000110e980 in io_handler_thread (arg=<optimized out>) at /export/home/pb2/build/sb_0-26514852-1514433850.9/mysql-5.7.21/storage/innobase/srv/srv0start.cc:311
#7  0x00007f50f13c1df5 in start_thread () from /lib64/libpthread.so.0
#8  0x00007f50efe7f1ad in clone () from /lib64/libc.so.6

//Omitted.

Thread 4 (Thread 0x7f50deeb2700 (LWP 3874)):
#0  0x00007f50f13c8ec1 in sigwait () from /lib64/libpthread.so.0
#1  0x00000000007c357b in signal_hand (arg=<optimized out>) at /export/home/pb2/build/sb_0-26514852-1514433850.9/mysql-5.7.21/sql/mysqld.cc:2120
#2  0x0000000000f707b4 in pfs_spawn_thread (arg=0x386a740) at /export/home/pb2/build/sb_0-26514852-1514433850.9/mysql-5.7.21/storage/perfschema/pfs.cc:2190
#3  0x00007f50f13c1df5 in start_thread () from /lib64/libpthread.so.0
#4  0x00007f50efe7f1ad in clone () from /lib64/libc.so.6

Thread 3 (Thread 0x7f50be7fc700 (LWP 3875)):
#0  0x00007f50f13c5705 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x0000000000ebe6f5 in native_cond_wait (mutex=<optimized out>, cond=<optimized out>) at /export/home/pb2/build/sb_0-26514852-1514433850.9/mysql-5.7.21/include/thr_cond.h:140
#2  my_cond_wait (mp=<optimized out>, cond=<optimized out>) at /export/home/pb2/build/sb_0-26514852-1514433850.9/mysql-5.7.21/include/thr_cond.h:195
#3  inline_mysql_cond_wait (src_line=874, src_file=0x15d92e0 "/export/home/pb2/build/sb_0-26514852-1514433850.9/mysql-5.7.21/sql/rpl_gtid_persist.cc", mutex=<optimized out>, that=<optimized out>) at /export/home/pb2/build/sb_0-26514852-1514433850.9/mysql-5.7.21/include/mysql/psi/mysql_thread.h:1184
#4  compress_gtid_table (p_thd=<optimized out>) at /export/home/pb2/build/sb_0-26514852-1514433850.9/mysql-5.7.21/sql/rpl_gtid_persist.cc:874
#5  0x0000000000f707b4 in pfs_spawn_thread (arg=0x386a740) at /export/home/pb2/build/sb_0-26514852-1514433850.9/mysql-5.7.21/storage/perfschema/pfs.cc:2190
#6  0x00007f50f13c1df5 in start_thread () from /lib64/libpthread.so.0
#7  0x00007f50efe7f1ad in clone () from /lib64/libc.so.6

Thread 2 (Thread 0x7f50dee81700 (LWP 3981)):
#0  0x00007f50efe74b7d in poll () from /lib64/libc.so.6
#1  0x00000000012bae8f in vio_io_wait (vio=<optimized out>, event=<optimized out>, timeout=300000) at /export/home/pb2/build/sb_0-26514852-1514433850.9/mysql-5.7.21/vio/viosocket.c:786
#2  0x00000000012baf83 in vio_socket_io_wait (vio=<optimized out>, event=<optimized out>) at /export/home/pb2/build/sb_0-26514852-1514433850.9/mysql-5.7.21/vio/viosocket.c:77
#3  0x00000000012bb5b8 in vio_read (vio=0x7f50b0000bb0, buf=0x7f50b000a3c0 "\001", size=4) at /export/home/pb2/build/sb_0-26514852-1514433850.9/mysql-5.7.21/vio/viosocket.c:132
#4  0x0000000000c6de03 in net_read_raw_loop (net=0x7f50b0002570, count=4) at /export/home/pb2/build/sb_0-26514852-1514433850.9/mysql-5.7.21/sql/net_serv.cc:672
#5  0x0000000000c6e69b in net_read_packet_header (net=0x7f50b0002570) at /export/home/pb2/build/sb_0-26514852-1514433850.9/mysql-5.7.21/sql/net_serv.cc:756
#6  net_read_packet (net=0x7f50b0002570, complen=0x7f50dee80d38) at /export/home/pb2/build/sb_0-26514852-1514433850.9/mysql-5.7.21/sql/net_serv.cc:822
#7  0x0000000000c6e94c in my_net_read (net=0x7f50b0002570) at /export/home/pb2/build/sb_0-26514852-1514433850.9/mysql-5.7.21/sql/net_serv.cc:899
#8  0x0000000000c7c1dc in Protocol_classic::read_packet (this=0x7f50b0001e18) at /export/home/pb2/build/sb_0-26514852-1514433850.9/mysql-5.7.21/sql/protocol_classic.cc:808
#9  0x0000000000c7af52 in Protocol_classic::get_command (this=0x7f50b0001e18, com_data=0x7f50dee80de0, cmd=0x7f50dee80e0c) at /export/home/pb2/build/sb_0-26514852-1514433850.9/mysql-5.7.21/sql/protocol_classic.cc:965
#10 0x0000000000d1cf47 in do_command (thd=0x7f50b0000dc0) at /export/home/pb2/build/sb_0-26514852-1514433850.9/mysql-5.7.21/sql/sql_parse.cc:938
#11 0x0000000000ded7ac in handle_connection (arg=<optimized out>) at /export/home/pb2/build/sb_0-26514852-1514433850.9/mysql-5.7.21/sql/conn_handler/connection_handler_per_thread.cc:300
#12 0x0000000000f707b4 in pfs_spawn_thread (arg=0x382fa60) at /export/home/pb2/build/sb_0-26514852-1514433850.9/mysql-5.7.21/storage/perfschema/pfs.cc:2190
#13 0x00007f50f13c1df5 in start_thread () from /lib64/libpthread.so.0
#14 0x00007f50efe7f1ad in clone () from /lib64/libc.so.6

Thread 1 (Thread 0x7f50f17e8740 (LWP 3817)):
#0  0x00007f50efe74b7d in poll () from /lib64/libc.so.6
#1  0x0000000000deeaf9 in Mysqld_socket_listener::listen_for_connection_event (this=0x30d96f0) at /export/home/pb2/build/sb_0-26514852-1514433850.9/mysql-5.7.21/sql/conn_handler/socket_connection.cc:852
#2  0x00000000007c89c4 in connection_event_loop (this=0x373e050) at /export/home/pb2/build/sb_0-26514852-1514433850.9/mysql-5.7.21/sql/conn_handler/connection_acceptor.h:66
#3  mysqld_main (argc=79, argv=0x2f897b8) at /export/home/pb2/build/sb_0-26514852-1514433850.9/mysql-5.7.21/sql/mysqld.cc:5122
#4  0x00007f50efdaaaf5 in __libc_start_main () from /lib64/libc.so.6
#5  0x00000000007be479 in _start ()
TS 788084395.1530408210 2018-07-01 03:23:30

[root@zlm1 03:37:50 ~]
#
 1 [root@zlm2 10:49:28 ~]
 2 #cat mysqladmin.log
 3 +-----------------------------------------------+--------------------------------------------------+
 4 | Variable_name                                 | Value                                            |
 5 +-----------------------------------------------+--------------------------------------------------+
 6 | Aborted_clients                               | 0                                                |
 7 | Aborted_connects                              | 0                                                |
 8 | Binlog_cache_disk_use                         | 0                                                |
 9 | Binlog_cache_use                              | 0                                                |
10 | Binlog_stmt_cache_disk_use                    | 0                                                |
11 | Binlog_stmt_cache_use                         | 0                                                |
12 | Bytes_received                                | 2142                                             |
13 | Bytes_sent                                    | 155447                                           |
14 | Com_admin_commands                            | 0                                                |
15 | Com_assign_to_keycache                        | 0                                                |
16 | Com_alter_db                                  | 0                                                |
17 | Com_alter_db_upgrade                          | 0                                                |
18 ...
19 
20 +-----------------------------------------------+--------------------------------------------------+
21 | Variable_name                                 | Value                                            |
22 +-----------------------------------------------+--------------------------------------------------+
23 | Aborted_clients                               | 0                                                |
24 | Aborted_connects                              | 0                                                |
25 | Binlog_cache_disk_use                         | 0                                                |
26 | Binlog_cache_use                              | 0                                                |
27 | Binlog_stmt_cache_disk_use                    | 0                                                |
28 | Binlog_stmt_cache_use                         | 0                                                |
29 | Bytes_received                                | 2177                                             |
30 | Bytes_sent                                    | 165275                                           |
31 | Com_admin_commands                            | 0                                                |
32 | Com_assign_to_keycache                        | 0                                                |
33 | Com_alter_db                                  | 0                                                |
34 | Com_alter_db_upgrade                          | 0                                                |
35 ...
36 
37 +-----------------------------------------------+--------------------------------------------------+
38 | Variable_name                                 | Value                                            |
39 +-----------------------------------------------+--------------------------------------------------+
40 | Aborted_clients                               | 0                                                |
41 | Aborted_connects                              | 0                                                |
42 | Binlog_cache_disk_use                         | 0                                                |
43 | Binlog_cache_use                              | 0                                                |
44 | Binlog_stmt_cache_disk_use                    | 0                                                |
45 | Binlog_stmt_cache_use                         | 0                                                |
46 | Bytes_received                                | 2212                                             |
47 | Bytes_sent                                    | 175103                                           |
48 | Com_admin_commands                            | 0                                                |
49 | Com_assign_to_keycache                        | 0                                                |
50 | Com_alter_db                                  | 0                                                |
51 | Com_alter_db_upgrade                          | 0                                                |

 

 

 

 

**Check the net relevant output by press
“q”,then press “n” in above output.**

**Check the value of “show global status
… “.**

**Demonstrate of relevant
conceptions.**

**Execute pt-mext to print them again
with the “mysqladmin.log” file.**

1 Press a key to continue or choose a different action
2 ======== zlm2 at 2018_07_09_08_19_39 NETWORK (1 of 2) ========
3 Source of connections to port 3306
4      12 192.168.1.101
5       2 
6 Status of connections to port 3306
7      12 ESTABLISHED
8       2 LISTEN
9 Press a key to continue or choose a different action
 1 (root@localhost mysql3306.sock)[sysbench]>show global status like '%Threads%';
 2 +------------------------+-------+
 3 | Variable_name          | Value |
 4 +------------------------+-------+
 5 | Delayed_insert_threads | 0     |
 6 | Slow_launch_threads    | 0     |
 7 | Threads_cached         | 1     |
 8 | Threads_connected      | 5     | //Threads_connected has been up to 5.
 9 | Threads_created        | 5     |
10 | Threads_running        | 1     |
11 +------------------------+-------+
12 6 rows in set (0.04 sec)
13 
14 
15 (root@localhost mysql3306.sock)[sysbench]>show global status like '%Threads%';
16 +------------------------+-------+
17 | Variable_name          | Value |
18 +------------------------+-------+
19 | Delayed_insert_threads | 0     |
20 | Slow_launch_threads    | 0     |
21 | Threads_cached         | 3     |
22 | Threads_connected      | 6     | //Second time,Threads_connected has been up to 6.
23 | Threads_created        | 9     |
24 | Threads_running        | 1     |
25 +------------------------+-------+
26 6 rows in set (0.01 sec)
 1 1.What's gdb?
 2 gdb is tool of linux,a debugger of GNU project.It can help you to see what's going on inside the program while it executes.For example,the usual case is the program running slowly or almost to be crashed.
 3 
 4 2.What can gdb do?
 5 It can catch details in the act belowl:
 6 1).Start the program and specify anything the might affect its behavior.
 7 2).Let the program stop on specific conditions.
 8 3).Check the details when the program stops.
 9 4).Change things in the program and try to correct effect of potential problems.
10 
11 3.What's LWP?
12 LWP is abbreviation of Light Weight Process.
13 Process:You'll find a process in background after you've executed a linux command(which can be observed by "ps -ef").
14 Thread:It's nothing but a flow of execution of the linux process.
15 Light Weight Process:It's almost the same with "thread" but a term at kernal level.
16 
17 Here's some detail of it:
18 What are Linux Processes, Threads, Light Weight Processes, and Process State
19 https://www.thegeekstuff.com/2013/11/linux-process-and-threads/ 
 1 [root@zlm2 10:56:56 ~]
 2 #pt-mext -- cat mysqladmin.log
 3 Aborted_clients                               0       0
 4 Aborted_connects                              0       0
 5 Binlog_cache_disk_use                         0       0
 6 Binlog_cache_use                              0       0
 7 Binlog_stmt_cache_disk_use                    0       0
 8 Binlog_stmt_cache_use                         0       0
 9 Bytes_received                             2142    2177
10 Bytes_sent                               155447  165275
11 Com_admin_commands                            0       0
12 Com_alter_db                                  0       0
13 Com_alter_db_upgrade                          0       0
14 ...
15 
16 //pt-mext arranges the results of those three times side-by-side and print them.
17 //The first column shows the first collected result and the second column shows the second one.
18 //The number of columns is n-1(n is the collection times here).

 

 

 

 

**Check the first four samples of “show
status” by press “m”.**

**Check the output files in default
directory(“/var/lib/pt-stalk”).**

**Something about output
file.**

**Specify the “–relative” option to get
the differential data.**

 1 ======== zlm2 at 2018_07_09_08_24_39 MEXT (2 of 2) ========
 2 Displaying the first 4 samples of SHOW STATUS counters
 3 ... //Several otmitted blank rows here.
 4 Aborted_clients                               3           0           0           0
 5 Aborted_connects                              0           0           0           0
 6 Binlog_cache_disk_use                         0           0           0           0
 7 Binlog_cache_use                         260350         275         262         286
 8 Binlog_stmt_cache_disk_use                    0           0           0           0
 9 Binlog_stmt_cache_use                        31           0           0           0
10 Bytes_received                        401068901      223306      212357      231011
11 Bytes_sent                          10433667031    11046014    10595838    11481359
12 Com_admin_commands                            2           0           0           0
13 Com_alter_db                                  0           0           0           0
14 Com_alter_db_upgrade                          0           0           0           0
15 Com_alter_event                               0           0           0           0
16 Com_alter_function                            0           0           0           0
17 Com_alter_instance                            0           0           0           0
18 Com_alter_procedure                           0           0           0           0
19 Com_alter_server                              0           0           0           0
20 Com_alter_table                               0           0           0           0
21 Com_alter_tablespace                          0           0           0           0
22 Com_alter_user                                1           0           0           0
23 Com_analyze                                   0           0           0           0
24 Com_assign_to_keycache                        0           0           0           0
25 Com_begin                                259973         274         263         286
26 ... //Omitted.
 1 [root@zlm2 08:31:41 /var/lib/pt-stalk]
 2 #ls -lrt
 3 total 3928
 4 -rw-r--r-- 1 root root     383 Jul  9 08:19 2018_07_09_08_19_39-trigger
 5 -rw-r--r-- 1 root root    9845 Jul  9 08:19 2018_07_09_08_19_39-pmap
 6 -rw-r--r-- 1 root root   16554 Jul  9 08:19 2018_07_09_08_19_39-variables
 7 -rw-r--r-- 1 root root    4352 Jul  9 08:19 2018_07_09_08_19_39-innodbstatus1
 8 -rw-r--r-- 1 root root     384 Jul  9 08:19 2018_07_09_08_19_39-mutex-status1
 9 -rw-r--r-- 1 root root    8507 Jul  9 08:19 2018_07_09_08_19_39-ps
10 -rw-r--r-- 1 root root   10356 Jul  9 08:19 2018_07_09_08_19_39-lsof
11 -rw-r--r-- 1 root root    5280 Jul  9 08:19 2018_07_09_08_19_39-opentables1
12 -rw-r--r-- 1 root root    8204 Jul  9 08:19 2018_07_09_08_19_39-top
13 -rw-r--r-- 1 root root   28343 Jul  9 08:19 2018_07_09_08_19_39-sysctl
14 -rw-r--r-- 1 root root     139 Jul  9 08:20 2018_07_09_08_19_39-disk-space
15 -rw-r--r-- 1 root root 1084769 Jul  9 08:20 2018_07_09_08_19_39-mysqladmin
16 -rw-r--r-- 1 root root    2782 Jul  9 08:20 2018_07_09_08_19_39-vmstat
17 -rw-r--r-- 1 root root   24750 Jul  9 08:20 2018_07_09_08_19_39-procstat
18 -rw-r--r-- 1 root root   14403 Jul  9 08:20 2018_07_09_08_19_39-diskstats
19 -rw-r--r-- 1 root root   72469 Jul  9 08:20 2018_07_09_08_19_39-procvmstat
20 -rw-r--r-- 1 root root   51960 Jul  9 08:20 2018_07_09_08_19_39-netstat_s
21 -rw-r--r-- 1 root root  296850 Jul  9 08:20 2018_07_09_08_19_39-slabinfo
22 -rw-r--r-- 1 root root   33210 Jul  9 08:20 2018_07_09_08_19_39-interrupts
23 -rw-r--r-- 1 root root   48390 Jul  9 08:20 2018_07_09_08_19_39-netstat
24 -rw-r--r-- 1 root root   37260 Jul  9 08:20 2018_07_09_08_19_39-meminfo
25 -rw-r--r-- 1 root root   17100 Jul  9 08:20 2018_07_09_08_19_39-df
26 -rw-r--r-- 1 root root   40753 Jul  9 08:20 2018_07_09_08_19_39-processlist
27 -rw-r--r-- 1 root root    4555 Jul  9 08:20 2018_07_09_08_19_39-innodbstatus2
28 -rw-r--r-- 1 root root   74476 Jul  9 08:20 2018_07_09_08_19_39-transactions
29 -rw-r--r-- 1 root root       5 Jul  9 08:20 2018_07_09_08_19_39-hostname
30 -rw-r--r-- 1 root root     385 Jul  9 08:20 2018_07_09_08_19_39-mutex-status2
31 -rw-r--r-- 1 root root    5280 Jul  9 08:20 2018_07_09_08_19_39-opentables2
32 -rw-r--r-- 1 root root     325 Jul  9 08:20 2018_07_09_08_19_39-vmstat-overall
33 -rw-r--r-- 1 root root   24763 Jul  9 08:20 2018_07_09_08_19_39-output
34 
35 //There're a series of files(starts with "tigger" and ends with "output") each time we collect the diagnostic data.
36 
37 -rw-r--r-- 1 root root     383 Jul  9 08:24 2018_07_09_08_24_39-trigger
38 -rw-r--r-- 1 root root   10251 Jul  9 08:24 2018_07_09_08_24_39-pmap
39 -rw-r--r-- 1 root root   16554 Jul  9 08:24 2018_07_09_08_24_39-variables
40 -rw-r--r-- 1 root root    4353 Jul  9 08:24 2018_07_09_08_24_39-innodbstatus1
41 -rw-r--r-- 1 root root    8583 Jul  9 08:24 2018_07_09_08_24_39-ps
42 -rw-r--r-- 1 root root     386 Jul  9 08:24 2018_07_09_08_24_39-mutex-status1
43 -rw-r--r-- 1 root root   28343 Jul  9 08:24 2018_07_09_08_24_39-sysctl
44 -rw-r--r-- 1 root root   10534 Jul  9 08:24 2018_07_09_08_24_39-lsof
45 -rw-r--r-- 1 root root    5280 Jul  9 08:24 2018_07_09_08_24_39-opentables1
46 -rw-r--r-- 1 root root    8053 Jul  9 08:24 2018_07_09_08_24_39-top
47 -rw-r--r-- 1 root root     139 Jul  9 08:25 2018_07_09_08_24_39-disk-space
48 -rw-r--r-- 1 root root    2783 Jul  9 08:25 2018_07_09_08_24_39-vmstat
49 -rw-r--r-- 1 root root 1084769 Jul  9 08:25 2018_07_09_08_24_39-mysqladmin
50 -rw-r--r-- 1 root root   24899 Jul  9 08:25 2018_07_09_08_24_39-procstat
51 -rw-r--r-- 1 root root   48390 Jul  9 08:25 2018_07_09_08_24_39-netstat
52 -rw-r--r-- 1 root root  296850 Jul  9 08:25 2018_07_09_08_24_39-slabinfo
53 -rw-r--r-- 1 root root   52230 Jul  9 08:25 2018_07_09_08_24_39-netstat_s
54 -rw-r--r-- 1 root root   33210 Jul  9 08:25 2018_07_09_08_24_39-interrupts
55 -rw-r--r-- 1 root root   37260 Jul  9 08:25 2018_07_09_08_24_39-meminfo
56 -rw-r--r-- 1 root root   14460 Jul  9 08:25 2018_07_09_08_24_39-diskstats
57 -rw-r--r-- 1 root root   17100 Jul  9 08:25 2018_07_09_08_24_39-df
58 -rw-r--r-- 1 root root   72914 Jul  9 08:25 2018_07_09_08_24_39-procvmstat
59 -rw-r--r-- 1 root root   44370 Jul  9 08:25 2018_07_09_08_24_39-processlist
60 -rw-r--r-- 1 root root    4445 Jul  9 08:25 2018_07_09_08_24_39-innodbstatus2
61 -rw-r--r-- 1 root root   76282 Jul  9 08:25 2018_07_09_08_24_39-transactions
62 -rw-r--r-- 1 root root       5 Jul  9 08:25 2018_07_09_08_24_39-hostname
63 -rw-r--r-- 1 root root     387 Jul  9 08:25 2018_07_09_08_24_39-mutex-status2
64 -rw-r--r-- 1 root root    5280 Jul  9 08:25 2018_07_09_08_24_39-opentables2
65 -rw-r--r-- 1 root root     325 Jul  9 08:25 2018_07_09_08_24_39-vmstat-overall
66 -rw-r--r-- 1 root root   24763 Jul  9 08:25 2018_07_09_08_24_39-output
67 
68 [root@zlm2 08:33:37 /var/lib/pt-stalk]
69 #cat 2018_07_09_08_24_39-trigger
70 2018_07_09_08_24_39 Check results: status(Threads_connected)=6, matched=yes, cycles_true=1
71 2018_07_09_08_24_39 pt-stalk ran with --function=status --variable=Threads_connected --threshold=2 --match= --cycles=1 --interval=1 --iterations= --run-time=30 --sleep=300 --dest=/var/lib/pt-stalk --prefix= --notify-by-email= --log=/var/log/pt-stalk.log --pid=/var/run/pt-stalk.pid --plugin=
72 
73 //The trigger file records the options we have used.
74 
75 [root@zlm2 08:34:17 /var/lib/pt-stalk]
76 #cat 2018_07_09_08_24_39-vmstat-overall
77 procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
78  r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
79 18  2   9692  60716      0 583316    0    0    11   139   92  597  4  1 96  0  0
80  4  0  10440  78828      0 569672    0   25    89  1965 1455 12046 80 20  0  0  0
81  
82  //The vmstat-overall records the vmstat output.
83  //each file contains the relevant diagnostic data about what their name called.I'm not going to demonstrate all of them.
1 1.The second column in the file shows the detail of memory address with hex.
2 2.At the very beginning of the file in Thread 1(LWP 3817),it shows the detail of how MySQL process works.
3 "Mysqld_socket_listener::listen_for_connection_event" is a MySQL function defined in source code file "socket_connection.cc".This is a clue which can help you to findout where to servay the details if something happened and stucked here.
4 3.At every bottom of each thread(except thread 1),you can see the same contents below:
5 "#. 0x... clone () from /lib64/libc.so.6" //clone() is a function to create a new thread.
6 "#. 0x... start_thread () from /lib64/libpthread.so.0" //start_thread() is a function to start a poxis thread.
 1 [root@zlm2 11:05:50 ~]
 2 #pt-mext --relative -- cat mysqladmin.log
 3 Aborted_clients                               0       0
 4 Aborted_connects                              0       0
 5 Binlog_cache_disk_use                         0       0
 6 Binlog_cache_use                              0       0
 7 Binlog_stmt_cache_disk_use                    0       0
 8 Binlog_stmt_cache_use                         0       0
 9 Bytes_received                             2142      35 //2277-2142=35
10 Bytes_sent                               155447    9828 //165275-155447=9828
11 Com_admin_commands                            0       0
12 Com_alter_db                                  0       0
13 Com_alter_db_upgrade                          0       0
14 
15 //Differential data is more distinct for us to know about the variation of performance.
16 //Even though we use "--relative" option,there're still two columns here.

 

 

 

 

**Check the file list by press “q”,then
press “0” in above output.**

**Take care of you remain disk
space,pt-stalk won’t run anymore if you’re out of
space.**

Summary

**997755.com澳门葡京,Execute pt-mext without flat text
file.**

 1 ======== zlm2 at 2018_07_09_08_19_39 LIST (1 of 2) ========
 2 -rw-r--r-- 1 root root  17K Jul  9 08:20 /var/lib/pt-stalk/2018_07_09_08_19_39-df
 3 -rw-r--r-- 1 root root  139 Jul  9 08:20 /var/lib/pt-stalk/2018_07_09_08_19_39-disk-space
 4 -rw-r--r-- 1 root root  15K Jul  9 08:20 /var/lib/pt-stalk/2018_07_09_08_19_39-diskstats
 5 -rw-r--r-- 1 root root    5 Jul  9 08:20 /var/lib/pt-stalk/2018_07_09_08_19_39-hostname
 6 -rw-r--r-- 1 root root 4.3K Jul  9 08:19 /var/lib/pt-stalk/2018_07_09_08_19_39-innodbstatus1
 7 -rw-r--r-- 1 root root 4.5K Jul  9 08:20 /var/lib/pt-stalk/2018_07_09_08_19_39-innodbstatus2
 8 -rw-r--r-- 1 root root  33K Jul  9 08:20 /var/lib/pt-stalk/2018_07_09_08_19_39-interrupts
 9 -rw-r--r-- 1 root root  11K Jul  9 08:19 /var/lib/pt-stalk/2018_07_09_08_19_39-lsof
10 -rw-r--r-- 1 root root  37K Jul  9 08:20 /var/lib/pt-stalk/2018_07_09_08_19_39-meminfo
11 -rw-r--r-- 1 root root  384 Jul  9 08:19 /var/lib/pt-stalk/2018_07_09_08_19_39-mutex-status1
12 -rw-r--r-- 1 root root  385 Jul  9 08:20 /var/lib/pt-stalk/2018_07_09_08_19_39-mutex-status2
13 -rw-r--r-- 1 root root 1.1M Jul  9 08:20 /var/lib/pt-stalk/2018_07_09_08_19_39-mysqladmin
14 -rw-r--r-- 1 root root  48K Jul  9 08:20 /var/lib/pt-stalk/2018_07_09_08_19_39-netstat
15 -rw-r--r-- 1 root root  51K Jul  9 08:20 /var/lib/pt-stalk/2018_07_09_08_19_39-netstat_s
16 -rw-r--r-- 1 root root 5.2K Jul  9 08:19 /var/lib/pt-stalk/2018_07_09_08_19_39-opentables1
17 -rw-r--r-- 1 root root 5.2K Jul  9 08:20 /var/lib/pt-stalk/2018_07_09_08_19_39-opentables2
18 -rw-r--r-- 1 root root  25K Jul  9 08:20 /var/lib/pt-stalk/2018_07_09_08_19_39-output
19 -rw-r--r-- 1 root root 9.7K Jul  9 08:19 /var/lib/pt-stalk/2018_07_09_08_19_39-pmap
20 -rw-r--r-- 1 root root  40K Jul  9 08:20 /var/lib/pt-stalk/2018_07_09_08_19_39-processlist
21 -rw-r--r-- 1 root root  25K Jul  9 08:20 /var/lib/pt-stalk/2018_07_09_08_19_39-procstat
22 -rw-r--r-- 1 root root  71K Jul  9 08:20 /var/lib/pt-stalk/2018_07_09_08_19_39-procvmstat
23 -rw-r--r-- 1 root root 8.4K Jul  9 08:19 /var/lib/pt-stalk/2018_07_09_08_19_39-ps
24 -rw-r--r-- 1 root root 290K Jul  9 08:20 /var/lib/pt-stalk/2018_07_09_08_19_39-slabinfo
25 -rw-r--r-- 1 root root  28K Jul  9 08:19 /var/lib/pt-stalk/2018_07_09_08_19_39-sysctl
26 -rw-r--r-- 1 root root 8.1K Jul  9 08:19 /var/lib/pt-stalk/2018_07_09_08_19_39-top
27 -rw-r--r-- 1 root root  73K Jul  9 08:20 /var/lib/pt-stalk/2018_07_09_08_19_39-transactions
28 -rw-r--r-- 1 root root  383 Jul  9 08:19 /var/lib/pt-stalk/2018_07_09_08_19_39-trigger
29 -rw-r--r-- 1 root root  17K Jul  9 08:19 /var/lib/pt-stalk/2018_07_09_08_19_39-variables
30 -rw-r--r-- 1 root root 2.8K Jul  9 08:20 /var/lib/pt-stalk/2018_07_09_08_19_39-vmstat
31 -rw-r--r-- 1 root root  325 Jul  9 08:20 /var/lib/pt-stalk/2018_07_09_08_19_39-vmstat-overall
32 
33 //It merely output the files relevant with the prefix("2018_07_09_08_19_39") you've specified.
 1 [root@zlm2 08:25:56 /data/mysql/mysql3306]
 2 #pt-stalk --host localhost --port 3306 --user root --password Passw0rd --collect --cycles 1 --variable Threads_connected --threshold 2 --sleep 30 --demonize
 3 Cannot open /tmp/pt-stalk.8940.FSboRq/po/daemonize: No space left on device at -e line 13, <$fh> chunk 274.
 4 No long attribute in option spec /tmp/pt-stalk.8940.FSboRq/po/ask-pass
 5 
 6 [root@zlm2 08:51:27 /data/mysql/mysql3306]
 7 #df -h
 8 ]Filesystem               Size  Used Avail Use% Mounted on
 9 /dev/mapper/centos-root  8.4G  8.4G   20K 100% / //The root of linux has been out of space.
10 devtmpfs                 488M     0  488M   0% /dev
11 tmpfs                    497M     0  497M   0% /dev/shm
12 tmpfs                    497M  6.6M  491M   2% /run
13 tmpfs                    497M     0  497M   0% /sys/fs/cgroup
14 /dev/sda1                497M  118M  379M  24% /boot
15 none                      87G   81G  5.9G  94% /vagrant
  • pt-pmp is read-only tool and it
    depends on the linux gdb to collect stack traces.
  • when pt-pmp works,it will attach gdb
    to the program you specified,what will probably freeze the program
    for a while.
  • It’s
    not recommend to run the tool on a very busy product system because
    of the potential freezing period and resource consumption.
  • As a
    matter of fact,you can also do the stack tracing stuff by pstack
    tool but that’s not today’s motif.
 1 [root@zlm2 11:07:16 ~]
 2 #pt-mext -r -- mysqladmin ext -i1 -c3
 3 Aborted_clients                               0       0
 4 Aborted_connects                              0       0
 5 Binlog_cache_disk_use                         0       0
 6 Binlog_cache_use                              0       0
 7 Binlog_stmt_cache_disk_use                    0       0
 8 Binlog_stmt_cache_use                         0       0
 9 Bytes_received                             2453      35
10 Bytes_sent                               185024    9828
11 Com_admin_commands                            0       0
12 Com_alter_db                                  0       0
13 Com_alter_db_upgrade                          0       0
14 ...
15 
16 [root@zlm2 11:13:28 ~]
17 #pt-mext -- mysqladmin ext -i1 -c3
18 Aborted_clients                               0       0
19 Aborted_connects                              0       0
20 Binlog_cache_disk_use                         0       0
21 Binlog_cache_use                              0       0
22 Binlog_stmt_cache_disk_use                    0       0
23 Binlog_stmt_cache_use                         0       0
24 Bytes_received                             2764    2799
25 Bytes_sent                               214601  224429
26 Com_admin_commands                            0       0
27 Com_alter_db                                  0       0
28 Com_alter_db_upgrade                          0       0
29 ...

 

 

 

 

**Check all the contents of files by
press “*” in above output.**

Summary

Summary

 1 ======== zlm2 at 2018_07_09_08_19_39 VIEW (1 of 2) ========
 2 Viewing all files //This time we'll check all the contents in each file.
 3 ... //Several otmitted blank rows here.
 4 TS 1531117181.030895281 2018-07-09 08:19:41
 5 Filesystem              1K-blocks     Used Available Use% Mounted on
 6 /dev/mapper/centos-root   8775680  7993600    782080  92% /
 7 devtmpfs                   499536        0    499536   0% /dev
 8 tmpfs                      508740        0    508740   0% /dev/shm
 9 tmpfs                      508740     6716    502024   2% /run
10 tmpfs                      508740        0    508740   0% /sys/fs/cgroup
11 /dev/sda1                  508588   120796    387792  24% /boot
12 none                     90264572 83357056   6907516  93% /vagrant
13 TS 1531117182.002834196 2018-07-09 08:19:42
14 Filesystem              1K-blocks     Used Available Use% Mounted on
15 /dev/mapper/centos-root   8775680  7993772    781908  92% /
16 devtmpfs                   499536        0    499536   0% /dev
17 tmpfs                      508740        0    508740   0% /dev/shm
18 tmpfs                      508740     6716    502024   2% /run
19 tmpfs                      508740        0    508740   0% /sys/fs/cgroup
20 /dev/sda1                  508588   120796    387792  24% /boot
21 none                     90264572 83357056   6907516  93% /vagrant
22 TS 1531117183.011493818 2018-07-09 08:19:43
23 Filesystem              1K-blocks     Used Available Use% Mounted on
24 /dev/mapper/centos-root   8775680  7993792    781888  92% /
25 devtmpfs                   499536        0    499536   0% /dev
26 tmpfs                      508740        0    508740   0% /dev/shm
27 tmpfs                      508740     6716    502024   2% /run
28 tmpfs                      508740        0    508740   0% /sys/fs/cgroup
29 /dev/sda1                  508588   120796    387792  24% /boot
30 none                     90264572 83357056   6907516  93% /vagrant
31 TS 1531117184.019711999 2018-07-09 08:19:44
32 Filesystem              1K-blocks     Used Available Use% Mounted on
33 /dev/mapper/centos-root   8775680  7993820    781860  92% /
34 devtmpfs                   499536        0    499536   0% /dev
35 /var/lib/pt-stalk/2018_07_09_08_19_39-df (file 1 of 30) //There're 30 files in all relative to the prefix.
  • pt-stalk
    is another tool in diagnosing system performance and similar with
    pt-pmp but not the same.
  • pt-stalk
    will generate many statistic files which can really help you
    collecting information in almost every aspect.
  • pt-stalk
    has four main triggers:gdb,oprofile,strace,tcpdump.It’s flexible to
    collect data with diffrent dimensionality.
  • pt-mext
    has only three options and it is rather easy to use.
  • pt-mext
    is a useful tool in anaylzing differential data of 
    global status of 
    MySQL.
  • pt-mext
    can be used not not only in “mysqladmin” program but also
    others.

 

 

 

Summary

  • pt-sift
    is a tool used to quikly sift contents of the files generated by
    pt-stalk 
    by variety.**
  • pt-sift
    can impove your efficiency in retrieving huge amount of files by
    specifying prefix.
  • pt-sift
    relies on pt-stalk and even downloads the necessary tool when using
    the function of them,such as pt-diskstats,pt-mext,etc.

 

相关文章

发表评论

电子邮件地址不会被公开。 必填项已用*标注

*
*
Website