Troubleshooting with the multipathd Interactive Console

Posted on May 17, 2011 by jlim0930
The multipathd -k command is an interactive interface to the multipathd daemon. Entering this command brings up an interactive multipath console. After entering this command, you can enter help to get a list of available commands, you can enter a interactive command, or you can enter CTRL-D to quit.
The multipathd interactive console can be used to troubleshoot problems you may be having with your system. For example, the following command sequence displays the multipath configuration, including the defaults, before exiting the console.
# multipathd -k
> > show config
> > CTRL-D
The following command sequence ensures that multipath has picked up any changes to the
multipath.conf,
# multipathd -k
> > reconfigure
> > CTRL-D
Use the following command sequence to ensure that the path checker is working properly.
# multipathd -k
> > show paths
> > CTRL-D
Posted in System Stuff | Tagged , , | Leave a comment

Prevent yum from updating RHEL to a new release

Posted on April 7, 2011 by jlim0930

Sometimes you will run into a scenario where you need to keep an OS at certain version due to vendor support. Such as EMC/IBM SAN powerpath etc.. You might also want to remain at a certain version due to change management policies.

In this example I like to use yum to keep my RHEL guest current with security and bug fixes. The problem I ran into is that Red Hat has released a new minor version of it’s Enterprise Linux 5.x, and so yum wants to update to it. Well have no fear, as there is one line you need to add to yum.conf to solve your problem.

# vi /etc/yum.conf

add this within [main]: exclude=redhat-release*

exit and save changes

Now you can run sudo yum update and you will still be at your RHEL 5.x version (issue cat /etc/redhat-release to double check)

Posted in System Stuff | Tagged , | Leave a comment

Packet Analyzer: 15 TCPDUMP Command Examples

Posted on December 27, 2010 by jlim0930

Thanks the geek stuff

tcpdump command is also called as packet analyzer.

tcpdump command will work on most flavors of unix operating system. tcpdump allows us to save the packets that are captured, so that we can use it for future analysis. The saved file can be viewed by the same tcpdump command. We can also use open source software like wireshark to read the tcpdump pcap files.

In this tcpdump tutorial, let us discuss some practical examples on how to use the tcpdump command.

1. Capture packets from a particular ethernet interface using tcpdump -i

When you execute tcpdump command without any option, it will capture all the packets flowing through all the interfaces. -i option with tcpdump command, allows you to filter on a particular ethernet interface.

$ tcpdump -i eth1
14:59:26.608728 IP xx.domain.netbcp.net.52497 > valh4.lell.net.ssh: . ack 540 win 16554
14:59:26.610602 IP resolver.lell.net.domain > valh4.lell.net.24151:  4278 1/0/0 (73)
14:59:26.611262 IP valh4.lell.net.38527 > resolver.lell.net.domain:  26364+ PTR? 244.207.104.10.in-addr.arpa. (45)

In this example, tcpdump captured all the packets flows in the interface eth1 and displays in the standard output.

Note: Editcap utility is used to select or remove specific packets from dump file and translate them into a given format.

2. Capture only N number of packets using tcpdump -c

When you execute tcpdump command it gives packets until you cancel the tcpdump command. Using -c option you can specify the number of packets to capture.

$ tcpdump -c 2 -i eth0
listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes
14:38:38.184913 IP valh4.lell.net.ssh > yy.domain.innetbcp.net.11006: P 1457255642:1457255758(116) ack 1561463966 win 63652
14:38:38.690919 IP valh4.lell.net.ssh > yy.domain.innetbcp.net.11006: P 116:232(116) ack 1 win 63652
2 packets captured
13 packets received by filter
0 packets dropped by kernel

The above tcpdump command captured only 2 packets from interface eth0.

Note: Mergecap and TShark: Mergecap is a packet dump combining tool, which will combine multiple dumps into a single dump file. Tshark is a powerful tool to capture network packets, which can be used to analyze the network traffic. It comes with wireshark network analyzer distribution.

3. Display Captured Packets in ASCII using tcpdump -A

The following tcpdump syntax prints the packet in ASCII.

$ tcpdump -A -i eth0
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes
14:34:50.913995 IP valh4.lell.net.ssh > yy.domain.innetbcp.net.11006: P 1457239478:1457239594(116) ack 1561461262 win 63652
E.....@.@..]..i...9...*.V...]...P....h....E...>{..U=...g.
......G..7\+KA....A...L.
14:34:51.423640 IP valh4.lell.net.ssh > yy.domain.innetbcp.net.11006: P 116:232(116) ack 1 win 63652
E.....@.@..\..i...9...*.V..*]...P....h....7......X..!....Im.S.g.u:*..O&....^#Ba...
E..(R.@.|.....9...i.*...]...V..*P..OWp........

Note: Ifconfig command is used to configure network interfaces

4. Display Captured Packets in HEX and ASCII using tcpdump -XX

Some users might want to analyse the packets in hex values. tcpdump provides a way to print packets in both ASCII and HEX format.

$tcpdump -XX -i eth0
18:52:54.859697 IP zz.domain.innetbcp.net.63897 > valh4.lell.net.ssh: . ack 232 win 16511
        0x0000:  0050 569c 35a3 0019 bb1c 0c00 0800 4500  .PV.5.........E.
        0x0010:  0028 042a 4000 7906 c89c 10b5 aaf6 0f9a  .(.*@.y.........
        0x0020:  69c4 f999 0016 57db 6e08 c712 ea2e 5010  i.....W.n.....P.
        0x0030:  407f c976 0000 0000 0000 0000            @..v........
18:52:54.877713 IP 10.0.0.0 > all-systems.mcast.net: igmp query v3 [max resp time 1s]
        0x0000:  0050 569c 35a3 0000 0000 0000 0800 4600  .PV.5.........F.
        0x0010:  0024 0000 0000 0102 3ad3 0a00 0000 e000  .$......:.......
        0x0020:  0001 9404 0000 1101 ebfe 0000 0000 0300  ................
        0x0030:  0000 0000 0000 0000 0000 0000            ............

5. Capture the packets and write into a file using tcpdump -w

tcpdump allows you to save the packets to a file, and later you can use the packet file for further analysis.

$ tcpdump -w 08232010.pcap -i eth0
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes
32 packets captured
32 packets received by filter
0 packets dropped by kernel

-w option writes the packets into a given file. The file extension should be .pcap, which can be read by any network protocol

analyzer.

6. Reading the packets from a saved file using tcpdump -r

You can read the captured pcap file and view the packets for analysis, as shown below.

$tcpdump -tttt -r data.pcap
2010-08-22 21:35:26.571793 00:50:56:9c:69:38 (oui Unknown) > Broadcast, ethertype Unknown (0xcafe), length 74:
        0x0000:  0200 000a ffff 0000 ffff 0c00 3c00 0000  ............<...
        0x0010:  0000 0000 0100 0080 3e9e 2900 0000 0000  ........>.).....
        0x0020:  0000 0000 ffff ffff ad00 996b 0600 0050  ...........k...P
        0x0030:  569c 6938 0000 0000 8e07 0000            V.i8........
2010-08-22 21:35:26.571797 IP valh4.lell.net.ssh > zz.domain.innetbcp.net.50570: P 800464396:800464448(52) ack 203316566 win 71
2010-08-22 21:35:26.571800 IP valh4.lell.net.ssh > zz.domain.innetbcp.net.50570: P 52:168(116) ack 1 win 71
2010-08-22 21:35:26.584865 IP valh5.lell.net.ssh > 11.154.12.255.netbios-ns: NBT UDP PACKET(137): QUERY; REQUEST; BROADC

7. Capture packets with IP address using tcpdump -n

In all the above examples, it prints packets with the DNS address, but not the ip address. The following example captures the packets and it will display the IP address of the machines involved.

$ tcpdump -n -i eth0
15:01:35.170763 IP 10.0.19.121.52497 > 11.154.12.121.ssh: P 105:157(52) ack 18060 win 16549
15:01:35.170776 IP 11.154.12.121.ssh > 10.0.19.121.52497: P 23988:24136(148) ack 157 win 113
15:01:35.170894 IP 11.154.12.121.ssh > 10.0.19.121.52497: P 24136:24380(244) ack 157 win 113

8. Capture packets with proper readable timestamp using tcpdump -tttt

$ tcpdump -n -tttt -i eth0

2010-08-22 15:10:39.162830 IP 10.0.19.121.52497 > 11.154.12.121.ssh: . ack 49800 win 16390
2010-08-22 15:10:39.162833 IP 10.0.19.121.52497 > 11.154.12.121.ssh: . ack 50288 win 16660
2010-08-22 15:10:39.162867 IP 10.0.19.121.52497 > 11.154.12.121.ssh: . ack 50584 win 16586

9. Read packets longer than N bytes

You can receive only the packets greater than n number of bytes using a filter ‘greater’ through tcpdump command

$ tcpdump -w g_1024.pcap greater 1024

10. Receive only the packets of a specific protocol type

You can receive the packets based on the protocol type. You can specify one of these protocols — fddi, tr, wlan, ip, ip6, arp, rarp, decnet, tcp and udp. The following example captures only arp packets flowing through the eth0 interface.

$ tcpdump -i eth0 arp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes
19:41:52.809642 arp who-has valh5.lell.net tell valh9.lell.net
19:41:52.863689 arp who-has 11.154.12.1 tell valh6.lell.net
19:41:53.024769 arp who-has 11.154.12.1 tell valh7.lell.net

11. Read packets lesser than N bytes

You can receive only the packets lesser than n number of bytes using a filter ‘less’ through tcpdump command

$ tcpdump -w l_1024.pcap  less 1024

12. Receive packets flows on a particular port using tcpdump port

If you want to know all the packets received by a particular port on a machine, you can use tcpdump command as shown below.

$ tcpdump -i eth0 port 22
19:44:44.934459 IP valh4.lell.net.ssh > zz.domain.innetbcp.net.63897: P 18932:19096(164) ack 105 win 71
19:44:44.934533 IP valh4.lell.net.ssh > zz.domain.innetbcp.net.63897: P 19096:19260(164) ack 105 win 71
19:44:44.934612 IP valh4.lell.net.ssh > zz.domain.innetbcp.net.63897: P 19260:19424(164) ack 105 win 71

13. Capture packets for particular destination IP and Port

The packets will have source and destination IP and port numbers. Using tcpdump we can apply filters on source or destination IP and port number. The following command captures packets flows in eth0, with a particular destination ip and port number 22.

$ tcpdump -w xpackets.pcap -i eth0 dst 10.181.140.216 and port 22

14. Capture TCP communication packets between two hosts

If two different process from two different machines are communicating through tcp protocol, we can capture those packets using tcpdump as shown below.

$tcpdump -w comm.pcap -i eth0 dst 16.181.170.246 and port 22

You can open the file comm.pcap using any network protocol analyzer tool to debug any potential issues.

15. tcpdump Filter Packets – Capture all the packets other than arp and rarp

In tcpdump command, you can give “and”, “or” and “not” condition to filter the packets accordingly.

$ tcpdump -i eth0 not arp and not rarp
20:33:15.479278 IP resolver.lell.net.domain > valh4.lell.net.64639:  26929 1/0/0 (73)
20:33:15.479890 IP valh4.lell.net.16053 > resolver.lell.net.domain:  56556+ PTR? 255.107.154.15.in-addr.arpa. (45)
20:33:15.480197 IP valh4.lell.net.ssh > zz.domain.innetbcp.net.63897: P 540:1504(964) ack 1 win 96
20:33:15.487118 IP zz.domain.innetbcp.net.63897 > valh4.lell.net.ssh: . ack 540 win 16486
20:33:15.668599 IP 10.0.0.0 > all-systems.mcast.net: igmp query v3 [max resp time 1s]
Posted in System Stuff | Tagged , , | Leave a comment

Oracle Database Startup and Shutdown Procedure

Posted on December 27, 2010 by jlim0930

thanks the geek stuff

How To Startup Oracle Database

1. Login to the system with oracle username

Typical oracle installation will have oracle as username and dba as group. On Linux, do su to oracle as shown below.

$ su - oracle

2. Connect to oracle sysdba

Make sure ORACLE_SID and ORACLE_HOME are set properly as shown below.

$ env | grep ORA
ORACLE_SID=DEVDB
ORACLE_HOME=/u01/app/oracle/product/10.2.0



You can connect using either “/ as sysdba” or an oracle account that has DBA privilege.

$ sqlplus '/ as sysdba'
SQL*Plus: Release 10.2.0.3.0 - Production on Sun Jan 18 11:11:28 2009
Copyright (c) 1982, 2006, Oracle.  All Rights Reserved.

Connected to:
Oracle Database 10g Enterprise Edition Release 10.2.0.3.0 - Production
With the Partitioning and Data Mining options
SQL>

3. Start Oracle Database

The default SPFILE (server parameter file) is located under $ORACLE_HOME/dbs. Oracle will use this SPFILE during startup, if you don’t specify PFILE.



Oracle will look for the parameter file in the following order under $ORACLE_HOME/dbs. If any one of them exist, it will use that particular parameter file.

  1. spfile$ORACLE_SID.ora
  2. spfile.ora
  3. init$ORACLE_SID.ora



Type “startup” at the SQL command prompt to startup the database as shown below.

SQL> startup
ORACLE instance started.

Total System Global Area  812529152 bytes
Fixed Size                  2264280 bytes
Variable Size             960781800 bytes
Database Buffers           54654432 bytes
Redo Buffers                3498640 bytes
Database mounted.
Database opened.
SQL>



If you want to startup Oracle with PFILE, pass it as a parameter as shown below.

SQL> STARTUP PFILE=/u01/app/oracle/product/10.2.0/dbs/init.ora

How To Shutdown Oracle Database

Following three methods are available to shutdown the oracle database:

  1. Normal Shutdown
  2. Shutdown Immediate
  3. Shutdown Abort

1. Normal Shutdown

During normal shutdown, before the oracle database is shut down, oracle will wait for all active users to disconnect their sessions. As the parameter name (normal) suggest, use this option to shutdown the database under normal conditions.

SQL> shutdown
Database closed.
Database dismounted.
ORACLE instance shut down.
SQL>

2. Shutdown Immediate

During immediate shutdown, before the oracle database is shut down, oracle will rollback active transaction and disconnect all active users. Use this option when there is a problem with your database and you don’t have enough time to request users to log-off.

SQL> shutdown immediate;
Database closed.
Database dismounted.
ORACLE instance shut down.
SQL>

3. Shutdown Abort

During shutdown abort, before the oracle database is shutdown, all user sessions will be terminated immediately. Uncomitted transactions will not be rolled back. Use this option only during emergency situations when the “shutdown” and “shutdown immediate” doesn’t work.

$ sqlplus '/ as sysdba'
SQL*Plus: Release 10.2.0.3.0 - Production on Sun Jan 18 11:11:33 2009
Copyright (c) 1982, 2006, Oracle.  All Rights Reserved.
Connected to an idle instance.

SQL> shutdown abort
ORACLE instance shut down.
SQL>
Posted in System Stuff | Tagged , , | Leave a comment

9 Linux ethtool Examples to Manipulate Ethernet Card (NIC Card)

Posted on December 27, 2010 by jlim0930

Thanks to the geek stuff

1. List Ethernet Device Properties

When you execute ethtool command with a device name, it displays the following information about the ethernet device.

# ethtool eth0
Settings for eth0:
        Supported ports: [ TP ]
        Supported link modes:   10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Full
        Supports auto-negotiation: Yes
        Advertised link modes:  10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Full
        Advertised auto-negotiation: Yes
        Speed: 100Mb/s
        Duplex: Full
        Port: Twisted Pair
        PHYAD: 1
        Transceiver: internal
        Auto-negotiation: on
        Supports Wake-on: d
        Wake-on: d
        Link detected: yes

This above ethtool output displays ethernet card properties such as speed, wake on, duplex and the link detection status. Following are the three types of duplexes available.

  • Full duplex : Enables sending and receiving of packets at the same time. This mode is used when the ethernet device is connected to a switch.
  • Half duplex : Enables either sending or receiving of packets at a single point of time. This mode is used when the ethernet device is connected to a hub.
  • Auto-negotiation : If enabled, the ethernet device itself decides whether to use either full duplex or half duplex based on the network the ethernet device attached to.

2. Change NIC Parameter Using ethtool Option -s autoneg

The above ethtool eth0 output displays that the “Auto-negotiation” parameter is in enabled state. You can disable this using autoneg option in the ethtool as shown below.

# ifdown eth0
    eth0      device: Broadcom Corporation NetXtreme II BCM5709 Gigabit Ethernet (rev 20)
    eth0      configuration: eth-bus-pci-0000:0b:00.0

# ethtool  -s eth0 autoneg off

# ethtool eth0
Settings for eth0:
        Supported ports: [ TP ]
        Supported link modes:   10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Full
        Supports auto-negotiation: Yes
        Advertised link modes:  Not reported
        Advertised auto-negotiation: No
        Speed: Unknown! (65535)
        Duplex: Unknown! (255)
        Port: Twisted Pair
        PHYAD: 1
        Transceiver: internal
        Auto-negotiation: off
        Supports Wake-on: g
        Wake-on: g
        Link detected: no
# ifup eth0

After the above change, you could see that the “link detection” value changed to down and auto-negotiation is in off state.

3. Change the Speed of Ethernet Device

Using ethtool you can change the speed of the ethernet device to work with the certain network devices, and the newly assign speed value should be within the limited capacity.

# ethtool -s eth0 speed 100 autoneg off

# ethtool eth0
Settings for eth0:
        Supported ports: [ TP ]
        Supported link modes:   10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Full
        Supports auto-negotiation: Yes
        Advertised link modes:  Not reported
        Advertised auto-negotiation: No
        Speed: Unknown! (65535)
        Duplex: Unknown! (255)
        Port: Twisted Pair
        PHYAD: 1
        Transceiver: internal
        Auto-negotiation: off
        Supports Wake-on: g
        Wake-on: g
        Link detected: no

Once you change the speed when the adapter is online, it automatically goes offline, and you need to bring it back online using ifup command.

# ifup eth0
    eth0      device: Broadcom Corporation NetXtreme II BCM5709 Gigabit Ethernet (rev 20)
    eth0      configuration: eth-bus-pci-0000:0b:00.0
Checking for network time protocol daemon (NTPD):                     running

# ethtool eth0
Settings for eth0:
        Supported ports: [ TP ]
        Supported link modes:   10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Full
        Supports auto-negotiation: Yes
        Advertised link modes:  Not reported
        Advertised auto-negotiation: No
        Speed: 100Mb/s
        Duplex: Full
        Port: Twisted Pair
        PHYAD: 1
        Transceiver: internal
        Auto-negotiation: off
        Supports Wake-on: g
        Wake-on: g
        Link detected: yes

As shown in the above output, the speed changed from 1000Mb/s to 100Mb/s and auto-negotiation parameter is unset.

To change the Maximum Transmission Unit (MTU), refer to our ifconfig examples article.

4. Display Ethernet Driver Settings

ethtool -i option displays driver version, firmware version and bus details as shown below.

# ethtool -i eth0
driver: bnx2
version: 2.0.1-suse
firmware-version: 1.9.3
bus-info: 0000:04:00.0

5. Display Auto-negotiation, RX and TX of eth0

View the autonegotiation details about the specific ethernet device as shown below.

# ethtool -a eth0
Pause parameters for eth0:
Autonegotiate:  on
RX:             on
TX:             on

6. Display Network Statistics of Specific Ethernet Device

Use ethtool -S option to display the bytes transfered, received, errors, etc, as shown below.

# ethtool -S eth0
NIC statistics:
     rx_bytes: 74356477841
     rx_error_bytes: 0
     tx_bytes: 110725861146
     tx_error_bytes: 0
     rx_ucast_packets: 104169941
     rx_mcast_packets: 138831
     rx_bcast_packets: 59543904
     tx_ucast_packets: 118118510
     tx_mcast_packets: 10137453
     tx_bcast_packets: 2221841
     tx_mac_errors: 0
     tx_carrier_errors: 0
     rx_crc_errors: 0
     rx_align_errors: 0
     tx_single_collisions: 0
     tx_multi_collisions: 0
     tx_deferred: 0
     tx_excess_collisions: 0
     tx_late_collisions: 0
     tx_total_collisions: 0
     rx_fragments: 0
     rx_jabbers: 0
     rx_undersize_packets: 0
     rx_oversize_packets: 0
     rx_64_byte_packets: 61154057
     rx_65_to_127_byte_packets: 55038726
     rx_128_to_255_byte_packets: 426962
     rx_256_to_511_byte_packets: 3573763
     rx_512_to_1023_byte_packets: 893173
     rx_1024_to_1522_byte_packets: 42765995
     rx_1523_to_9022_byte_packets: 0
     tx_64_byte_packets: 3633165
     tx_65_to_127_byte_packets: 51169838
     tx_128_to_255_byte_packets: 3812067
     tx_256_to_511_byte_packets: 113766
     tx_512_to_1023_byte_packets: 104081
     tx_1024_to_1522_byte_packets: 71644887
     tx_1523_to_9022_byte_packets: 0
     rx_xon_frames: 0
     rx_xoff_frames: 0
     tx_xon_frames: 0
     tx_xoff_frames: 0
     rx_mac_ctrl_frames: 0
     rx_filtered_packets: 14596600
     rx_discards: 0
     rx_fw_discards: 0

7. Troubleshoot the Ethernet Connection Issues

When there is a problem with the network connection, you might want to check (or change) the ethernet device parameters explained in the above examples, when you see following issues in the output of ethtool command.

  • Speed and Duplex value is shown as Unknown
  • Link detection value is shown as No

Upon successful connection, the three parameters mentioned above gets appropriate values. i.e Speed is assigned with known value, Duplex become either Full/Half, and the Link detection becomes Yes.

After the above changes, if the Link Detection still says “No”, check whether there are any issues in the cables that runs from the switch and the system, you might want to dig into that aspect further.

To capture and analyze packets from a specific network interface, use tcpdump utility.

8. Identify Specific Device From Multiple Devices (Blink LED Port of NIC Card)

Let us assume that you have a machine with four ethernet adapters, and you want to identify the physical port of a particular ethernet card. (For example, eth0).

Use ethtool option -p, which will make the corresponding LED of physical port to blink.

# ethtool -p eth0

9. Make Changes Permanent After Reboot

If you’ve changed any ethernet device parameters using the ethtool, it will all disappear after the next reboot, unless you do the following.

On ubuntu, you have to modify /etc/network/interfaces file and add all your changes as shown below.

# vim /etc/network/interfaces
post-up ethtool -s eth2 speed 1000 duplex full autoneg off

The above line should be the last line of the file. This will change speed, duplex and autoneg of eth2 device permanently.

On SUSE, modify the /etc/sysconfig/network/ifcfg-eth-id file and include a new script using POST_UP_SCRIPT variable as shown below. Include the below line as the last line in the corresponding eth1 adpater config file.

# vim /etc/sysconfig/network/ifcfg-eth-id
POST_UP_SCRIPT='eth1'

Then, create a new file scripts/eth1 as shown below under /etc/sysconfig/network directory. Make sure that the script has execute permission and ensure that the ethtool utility is present under /sbin directory.

# cd /etc/sysconfig/network/

# vim scripts/eth1
#!/bin/bash
/sbin/ethtool -s duplex full speed 100 autoneg off
Posted in System Stuff | Tagged , , | Leave a comment