1、08/25/1975,Utilization of Software Tool “WireShark“ to Trouble Shoot TEC and other Application issues,2,Calvin Moody and Roy Spencer,08/25/2009,Why would you ever need to trace the network?,Although this never happens, we can get into a situation where a client application or vendor is trying to “bl
2、ame” us for a certain behavior The network tracing can give us additional insight to help with problem determination and resolution Avoid “Finger Pointing” and get to the root of the problem!,3,Calvin Moody and Roy Spencer,08/25/2009,Presentation Objectives,Overview of TCP stack and a typical TCP/IP
3、 packet exchange UDP vs TCP/IP, the OSI model, IPs, Sockets and Interfaces Methods/tools/execution for network tracing Making sense of the gathered trace.,4,Calvin Moody and Roy Spencer,08/25/2009,What is TCP/IP,TCP/IP = Transmission Control Protocol Internet Protocol. Originally invented by the DOD
4、 as a method for computers from different manufactures to communicate with one another, the protocol has become the standard for the majority of network based communication. The protocol works on the concept of “guaranteed deliverability” meaning that each packet transmitted contains a checksum cont
5、rol which requires acknowledgement from the remote receiving host. Should the transmitting system not receive the acknowledgement the last packet transmitted will be resent thus assuring delivery of the packet.,5,Calvin Moody and Roy Spencer,08/25/2009,A Basic TCP/IP example,TCP/IP is a method for c
6、omputer systems from different manufactures to talk to each other Uses a protocol standard to define the conversation The ability to analyze these conversations will help us become more effective in our troubleshooting,6,Calvin Moody and Roy Spencer,08/25/2009,An Overview of the typical TCP/UDP exch
7、ange,I have found this diagram to be the best representation of the overall picture TCP is “guaranteed packet delivery” UDP is something of a spray,7,Calvin Moody and Roy Spencer,08/25/2009,No network presentation is complete without,The OSI Model Application (our products) Presentation (how the pro
8、duct will talk) Session On which ports, sockets, speed etc Transport the protocol used (tcp, ldap etc) Network Physically sending/recving data Data well The data Physical Nic cards, Hubs, Routers, firewalls etc.,8,Calvin Moody and Roy Spencer,08/25/2009,Understanding the state of a socket Client sid
9、e,Socket States from the client perspective:,9,Calvin Moody and Roy Spencer,08/25/2009,Understanding the state of a socket Sever side,This is the state you can see while examining sockets from the Server side of the connection,10,Calvin Moody and Roy Spencer,08/25/2009,The entire picture Stephens di
10、agram,Calvin Moody and Roy Spencer,08/25/2009,Problem: Attempt to connect to incorrect port,problem:tn 9.48.205.199 5539Trying. telnet: connect: A remote host refused an attempted connect operation.trace from client system system outside of the firewall:iptrace -a -b -d 9.48.205.199 /tmp/client_iptr
11、ace1.binWhat takes place: ping 9.48.205.199tn 9.48.205.199 5539 Trying. telnet: connect: A remote host refused an attempted connect operation.,tn 9.48.205.199 . AIX Version 5 (C) Copyrights by IBM and by others 1982, 2007. login: CConnection closed.iptrace -u- I then talk to TEC admin and find out t
12、hat the port being used is actually in a LISTEN state is port 5529 as indicated by rpcinfo output NOTE: tec_reception is always going to be program id 100033057 while tec_rule is 100033058rpcinfo -p | grep 100033,Calvin Moody and Roy Spencer,08/25/2009,Problem: Attempt to connect to incorrect port,N
13、ext to view a good connection: client runsiptrace -a -b -d 9.48.205.199 /tmp/client_iptrace2.binserver runs iptrace -a -p 5529 /tmp/server_iptrace2.bintn 9.48.205.199 5529 Trying. Connected to 9.48.205.199.-indicating a open port which is LISTENing Escape character is T. (ctrl +c to break)duplicated
14、 again this time I killed the tn session as root:ps -ef |grep tnroot 970852 807030 0 09:35:09 pts/2 0:00 grep tnroot 999602 1011824 0 09:34:50 pts/0 0:00 tn 9.48.205.199 5529kill -9 999602 (still no RST).indicating a good connection.,program vers proto port service100033057 1 tcp 5529100033058 1 tcp
15、 5532Further verification can be done using netstat in conjunction with rmsock or stat -Aan | grep 5529 72575a10 tcp4 0 0 *.5529 *.* LISTEN 2 ROOTCENTAURI / rmsock 72575a10 tcpcb The socket 0x72575808 is being held by proccess 86494 (tec_reception).,Calvin Moody and Roy Spencer,08/25/2009,Wireshark
16、view of RST on incorrect Port 5539,Calvin Moody and Roy Spencer,08/25/2009,Wireshark view of good connection on Port 5529,15,Calvin Moody and Roy Spencer,08/25/2009,How do I test the socket state?,For example, if I wanted to see all the connections on my system for TECs LISTENer, tec_reception which
17、 runs on port 5529 (in this example) on UNIX I run: netstat -Aan | grep 5529 72575a10 tcp4 0 0 *.5529 *.* LISTEN 71dbb210 tcp4 0 0 9.48.205.199.5529 9.41.132.129.60880 ESTABLISHEDThis indicates I have a process running on port ports 5529 and 60880. I do have any established TEC connections.,16,Calvi
18、n Moody and Roy Spencer,08/25/2009,Expanding a little on netstat an output,There is a great deal of information that can be gleaned from the sockets: Proto Recv-Q Send-Q Local Address Foreign Address (state) tcp 0 0 *.23 *.* LISTEN tcp4 0 0 127.0.0.1.3827 127.0.0.1.37603 ESTABLISHED tcp4 0 0 9.41.13
19、2.129.32771 9.41.132.129.703 ESTABLISHED tcp4 0 0 9.41.132.129.703 9.41.132.129.32771 ESTABLISHEDProto: This is the protocol being used (TCP4/TCP6/UDP etc) Recv-Q: is the TCP buffer where data is stored prior to going up to the application layer. Data in this buffer may indicate an app layer hang Se
20、nd-Q: The same as the recv-q but this is transmit. This may indicate the remote side is having issue retrieving the data Local Address: almost always local host Foreign Address: the ip of the client. This is helpful if you are trying to identify a specific client system or which port number that cli
21、ent system is using. State: These vary and are defined in slides 8 and 9,17,Calvin Moody and Roy Spencer,08/25/2009,Some iptracing general best practices,1. Even non-production networks are VERY busy. Systems constantly keep in contact updating ARP tables, DNS and other communications - ALWAYS limit
22、 the scope of your tracing! 2. Target your tracing approach to the situation 3. There is a measurable performance hit associated with iptrace, and traces grow VERY fast. 4. Without a timeframe of reference an iptrace is generally a nightmare (avoid needle in the haystack cases iptrace is a supplemen
23、t to your normal logs and troubleshooting techniques. Align times in logs/traces to the timeframe in iptrace),18,Calvin Moody and Roy Spencer,08/25/2009,Tracing facilities on some of our common OS types:,19,Calvin Moody and Roy Spencer,08/25/2009,Methods and tools - Scope,Why do we limit the scope?
24、In this basic example I took a wide open iptrace on my mostly idle AIX system: #iptrace /opt/full_iptrc.bin (basically getting every packet of every protocol to and from this system) After letting this trace run for only 2 min the file size was already: -rw-r-r- 1 root system 19029 Aug 05 16:02 full
25、ip_trc.bin Although this might not seem very large, the point is that without scope the trace can rapidly go to large to be of any use.,20,Calvin Moody and Roy Spencer,08/25/2009,Limiting the scope,In this next example I want to only capture traffic to a specific port on my system (in this case ftp
26、traffic) #iptrace a p 21 ftp_iptrc.bin (in this case a remote system attempts to login with an invalid password) -rw-r-r- 1 root system 1566 Aug 05 16:07 ftp_iptrc.bin -rw-r-r- 1 root system 19029 Aug 05 16:02 full_trc.bin As you can see, by simply limiting ARP (-a) and telling the iptrace to watch
27、a specific port (-p) we have greatly limited the size of the binary output.,21,Calvin Moody and Roy Spencer,08/25/2009,A quick comment on SSL,Although iptrace will show the same TCP traffic, the data payload of the packet will be encrypted IPtrace is not very helpful when debugging application layer
28、 issues if SSL is being used. May want to include this for GSKIT team along with gskit tracing.,22,Calvin Moody and Roy Spencer,08/25/2009,Methods General Overview,Try and formulate a specific target problem (determine scope) Clear all logs Begin iptrace facility Duplicate the problem as quickly as
29、possible Terminate iptrace facility Gather logs and corresponding iptrace Evaluate trace with reporting utility or Wireshark,23,Calvin Moody and Roy Spencer,08/25/2009,Specific examples of limited scope iptraces on AIX,To capture all traffic to and from 9.128.1.4 (while suppressing arp packets): # i
30、ptrace a b d 9.128.1.4 /filesys/iptrace1.bin To capture all traffic on port 5539: #iptrace a p 5529 /filesys/iptrace2.bin To capture a rolling log of all port 5539 traffic: #iptrace a p 5529 L 500000 /filesys/iptrace3.bin,24,Calvin Moody and Roy Spencer,08/25/2009,Specific examples of limited scope
31、iptraces on Linux,In Linux we use tcpdump: To capture all traffic on eth0 suppressing arp: # tcpdump -ennqti eth0 (arp) /filesys/tcpdump1.bin To capture all traffic on port 5529: #tcpdump port 5529 The tcpdump is very robust (see man page) but this site has a few good usage examples: http:/www.erg.a
32、bdn.ac.uk/users/alastair/tcpdump.html#use,25,Calvin Moody and Roy Spencer,08/25/2009,Specific examples of limited scope iptraces on SUN,The utility to gather a network trace in Sun is the snoop command. To capture all traffic to and from 9.128.1.4#snoop o /filesystem/snoop1.bin 9.128.1.4, localhost
33、To capture all traffic on port 5529: #snoop o /filesystem/snoop2.bin port 5529,26,Calvin Moody and Roy Spencer,08/25/2009,We have the network trace now what?,In the previous slides we saw a few examples of how to collect the network trace Now we will talk about the tools we can use to analyze the bi
34、nary data gathered by the trace: Most iptraces are in binary format (non-human readable) Most can be read via Wireshark,27,Calvin Moody and Roy Spencer,08/25/2009,In order to begin we have to have,A basic understanding of the structure of a packet (TCP/IP in this example),28,Calvin Moody and Roy Spe
35、ncer,08/25/2009,What this packet structure would look like,In slide 16 we took an iptrace on AIX of a failing FTP login I used the ipreport command on AIX to make this readable: #ipreport rnNs ftp_iptrc.bin ftp_ipreport.out-r: know about rpc -n: number packets -N: dont do name resolution -s: start l
36、ines with protocol indicator strings,29,Calvin Moody and Roy Spencer,08/25/2009,The first packet will look something like:,Packet Number 1 ETH: =( 62 bytes received on interface en0 )= 16:06:49.514567639 ETH: 42:09:29:84:03:00 - 00:09:6b:2e:6f:4f type 800 (IP) IP: IP: IP: ip_v=4, ip_hl=20, ip_tos=0,
37、 ip_len=48, ip_id=50305, ip_off=0 DF IP: ip_ttl=119, ip_sum=235d, ip_p = 6 (TCP) TCP: TCP: th_seq=2976755260, th_ack=0 TCP: th_off=7, flags TCP: th_win=65535, th_sum=f554, th_urp=0 TCP: mss 1322 TCP: nop TCP: nop TCP: SACK PERMITTED,30,Calvin Moody and Roy Spencer,08/25/2009,TCP Flags a basic overvi
38、ew,31,Calvin Moody and Roy Spencer,08/25/2009,TCP Flags continued,You may see a variety of flags in the trace including: urg: tcpFlags = 0b100000! - Urgent Pointer field significant ack:tcpFlags = 0b010000! - Acknowledgment field significant psh:tcpFlags = 0b001000! - Push Function rst:tcpFlags = 0b
39、000100! - Reset the connection syn:tcpFlags = 0b000010! - Synchronize sequence numbers fin:tcpFlags = 0b000001! - No more data from sender,The RST (or RESET) flag is always what I look for when I have lost a normal connection. The RST means that during a normal TCP conversation something went south
40、and the connection was RESET (meaning the entire conversation would have to be redone starting at the SYN packet),32,Calvin Moody and Roy Spencer,08/25/2009,Why use text when you can use a GUI for free?,Along with ipreport in AIX there are other native utilities in most platforms to do some form of
41、reporting based on network trace However, the easiest tool for evaluating a network trace is the Wireshark utility http:/www.wireshark.org Its “award winning”!,33,Calvin Moody and Roy Spencer,08/25/2009,A quick point on transport of the network trace,These files are BINARY, and as such can be corrup
42、ted if you FTP in asci mode The files are typically large, I suggest a tar/zip of the original file (again making certain of binary mode file transfer) I suggest a min 2 GB RAM on your laptop to read very large trace,34,Calvin Moody and Roy Spencer,08/25/2009,Using Wireshark,Once downloaded and installed its like any other app,Calvin Moody and Roy Spencer,08/25/2009,Using Wireshark to extract one conversation.,Calvin Moody and Roy Spencer,08/25/2009,Follow TCP stream on port 5529,37,Calvin Moody and Roy Spencer,08/25/2009,Thank you for your time and attentionQUESTIONS,
