RHEL4:tg3,bond

Linux

Broadcom NICはbcm5700.koからtg3.koへ

  • bcm5700はメンテナンス終了(2006年中まで修正のみ対応)
  • bcm5700/baspの組み合わせからtg3/bondの組み合わせに
  • tg3
    • tg3はデフォルトでLinuxカーネルに組み込まれているが、IBMサーバで使用する場合はIBMから提供されているBroadcomドライバCDに含まれるtg3ドライバに変更して使用する
  1. Broadcom CDからtg3ドライバ(tg3-3.43b.tgz)をコピー
  2. 解凍してmake
  3. ネットワーク停止(service network stop)
  4. デフォルトで含まれているtg3ドライバをアンロード(rmmod tg3)
  5. makeしたドライバをロード(insmod ***/tg3.ko)
  6. ifconfigでロードされていることを確認
  7. 既存ドライバのバックアップ(/lib/modules/2.6.9-11.ELsmp/kernel/drivers/net/tg3.ko)
  8. ドライバをインストール(tg3ドライバを解凍したフォルダでmake install)
  9. modprobe.conf確認(alias eth0 tg3 ...)
  10. ドライバが正しく読み込まれることを確認(rmmod tg3 → service network start → lsmod)
    • パラメータの設定はethtoolコマンドを使用して実施する

[root@HS20-7981 ~]# ethtool eth0
Settings for eth0:
Supported ports: [ FIBRE ]
Supported link modes: 1000baseT/Half 1000baseT/Full
Supports auto-negotiation: Yes
Advertised link modes: 1000baseT/Half 1000baseT/Full
Advertised auto-negotiation: Yes
Speed: 1000Mb/s
Duplex: Full
Port: Twisted Pair
PHYAD: 1
Transceiver: internal
Auto-negotiation: on
Supports Wake-on: g
Wake-on: d
Current message level: 0x000000ff (255)
Link detected: yes

      • 詳細はman ethtoolで

[root@HS20-7981 mptsas]# ethtool
ethtool version 1.8
Usage:
ethtool DEVNAME
ethtool -a DEVNAME
ethtool -A DEVNAME \
[ autoneg on|off ] \
[ rx on|off ] \
[ tx on|off ]
ethtool -c DEVNAME
ethtool -C DEVNAME \
[adaptive-rx on|off] \
[adaptive-tx on|off] \
[rx-usecs N] \
[rx-frames N] \
[rx-usecs-irq N] \
[rx-frames-irq N] \
[tx-usecs N] \
[tx-frames N] \
[tx-usecs-irq N] \
[tx-frames-irq N] \
[stats-block-usecs N] \
[pkt-rate-low N] \
[rx-usecs-low N] \
[rx-frames-low N] \
[tx-usecs-low N] \
[tx-frames-low N] \
[pkt-rate-high N] \
[rx-usecs-high N] \
[rx-frames-high N] \
[tx-usecs-high N] \
[tx-frames-high N] \
[sample-interval N]
ethtool -g DEVNAME
ethtool -G DEVNAME \
[ rx N ] \
[ rx-mini N ] \
[ rx-jumbo N ] \
[ tx N ]
ethtool -i DEVNAME
ethtool -d DEVNAME
ethtool -e DEVNAME \
[ raw on|off ] \
[ offset N ] \
[ length N ]
ethtool -E DEVNAME \
[ magic N ] \
[ offset N ] \
[ value N ]
ethtool -k DEVNAME
ethtool -K DEVNAME \
[ rx on|off ] \
[ tx on|off ] \
[ sg on|off ] \
[ tso on|off ]
ethtool -r DEVNAME
ethtool -p DEVNAME [ %d ]
ethtool -t DEVNAME [online|(offline)]
ethtool -s DEVNAME \
[ speed 10|100|1000 ] \
[ duplex half|full ] \
[ port tp|aui|bnc|mii|fibre ] \
[ autoneg on|off ] \
[ phyad %d ] \
[ xcvr internal|external ] \
[ wol p|u|m|b|a|g|s|d... ] \
[ sopass %x:%x:%x:%x:%x:%x ] \
[ msglvl %d ]
ethtool -S DEVNAME

  • Teamingもbaspからbondingに変更
  • VLANもbaspからvconfigに変更

[root@HS20-7981 ~]# vconfig
Expecting argc to be 3-5, inclusive. Was: 1

Usage: add [interface-name] [vlan_id]
rem [vlan-name]
set_flag [interface-name] [flag-num] [0 | 1]
set_egress_map [vlan-name] [skb_priority] [vlan_qos]
set_ingress_map [vlan-name] [skb_priority] [vlan_qos]
set_name_type [name-type]

*The [interface-name] is the name of the ethernet card that hosts
the VLAN you are talking about.
*The vlan_id is the identifier (0-4095) of the VLAN you are operating on.
*skb_priority is the priority in the socket buffer (sk_buff).
*vlan_qos is the 3 bit priority in the VLAN header
*name-type: VLAN_PLUS_VID (vlan0005), VLAN_PLUS_VID_NO_PAD (vlan5),
DEV_PLUS_VID (eth0.0005), DEV_PLUS_VID_NO_PAD (eth0.5)
*bind-type: PER_DEVICE # Allows vlan 5 on eth0 and eth1 to be unique.
PER_KERNEL # Forces vlan 5 to be unique across all devices.
*FLAGS: 1 REORDER_HDR When this is set, the VLAN device will move the
ethernet header around to make it look exactly like a real
ethernet device. This may help programs such as DHCPd which
read the raw ethernet packet and make assumptions about the
location of bytes. If you don't need it, don't turn it on, because
there will be at least a small performance degradation. Default
is OFF.

  • ethtoolを使って起動時から設定を変更する場合は/etc/modprobe.confに書き込むこともできるが、その方法では複数NICの場合上手く動作しないので/etc/rc.localに記述する
  • /etc/modprobe.confに設定する場合のサンプル

alias eth0 tg3
post-install tg3 /sbin/ethtool -s eth0 speed 100 duplex full autoneg off

  • /etc/rc.localに設定する場合のサンプル

/sbin/ethtool -s eth0 speed 100 duplex full autoneg off

  • baspではTeamingに組み込むことが出来るNIC数に制限があったが、bondingでは基本的に制限はない
  • bondingによるTeaming構成手順
    1. ifcfg-eth*の設定
    2. bond*の設定
    3. modprobe.confの設定

設定例

  • ifcfg-eth0

DEVICE=eth0
BOOTPROTO=none
MASTEZR=bond0
SLAVE=yes

  • ifcfg-eth1

DEVICE=eth1
BOOTPROTO=none
MASTEZR=bond0
SLAVE=yes

  • ifcfg-eth2

DEVICE=eth2
BOOTPROTO=none
MASTEZR=bond1
SLAVE=yes

  • ifcfg-eth3

DEVICE=eth3
BOOTPROTO=none
MASTEZR=bond1
SLAVE=yes

  • ifcfg-bond0

DEVICE=bond0
IPADDR=*.*.*.*
NETMASK=*.*.*.*
NETWORK=*.*.*.*
BROADCAST=*.*.*.*
ONBOOT=yes
BOOTPROTO=static

  • ifcfg-bond1

DEVICE=bond1
IPADDR=*.*.*.*
NETMASK=*.*.*.*
NETWORK=*.*.*.*
BROADCAST=*.*.*.*
ONBOOT=yes
BOOTPROTO=static

  • modprobe.conf

alias eth0 tg3
alias eth1 tg3
alias eth2 tg3
alias eth3 tg3
alias bond0 bonding
alias bond1 bonding
options bond0 mode=1 miimon=5000
install bond1 /sbin/modprobe bonding -o bonding1 mode=1 miimon=5000

  • 停止：ifconfig bond0 down
  • 開始：ifconfig bond0 up
  • 設定変更：modprobe.conf設定変更 → ifconfig bond* down → rmmod bonding → ifconfig bond* up
  • 確認：/proc/net/bondind/bond*
  • リンク監視：必須
    • arp_interval/arp_target_ip：指定IPとの疎通確認
    • miimon：NICのUp/Down検知
  • Primary NICを指定した場合は復旧時には自動フェイルバックする

[root@HS20-7981 ~]# cat /proc/net/bonding/bond0
Ethernet Channel Bonding Driver: v2.6.1 (October 29, 2004)

Bonding Mode: fault-tolerance (active-backup)
Primary Slave: eth0
Currently Active Slave: None
MII Status: down
MII Polling Interval (ms): 5000
Up Delay (ms): 0
Down Delay (ms): 0