Convergence and Security on the Network (Jeffery Carrell)

• EAP is the layer two part, until you authenticate with EAP the port won't allow any layer three traffic.
• Most VOIP phones and some network printers have built in support for 802.1x (but often only weaker EAP methods).
• If you are assigning VLANs from Radius the best plan is to configure the client ports on a dead VLAN (eg. no access to anything), then once the client is authenticated it will add the port as an untagged VLAN.
• If you aren't assigning VLANs from Radius then you can configure the switch to change the ports VLAN on a successful authentication.
• Originally 802.1x made no provision for handing out tagged VLANs from Radius, this is now supported through RFC4675 (still not widely supported though).
• You can do all the same VLAN provisioning with MAC based authorisation instead of full user/pass authentication.
• Apparently WPA2 supports a non-shared key method which isn't 802.1x ... investigate!

HP Integrated Citrix XenServer on HP Proliant Servers (Chris Lynch, Brian Taylor & Aaron Olbrych)

• HP has their on version of XenServer caled "HP Select"
• "HP Select" integrates with Proliant virtual console so you can get "KVM" access to your VMs
• "PV Guest" = paravirtualised OS (modified kernel)
• "HVM Guest" = hardware-virtualised OS (non-modified kernel, requires Intel VT or AMD-V chipsets)
• SMP (Server Migration Pack) v3.5 supports XenServer (physical/virtual to physical/virtual on Proliant hardwre)
• Blah blah marketing blah blah.

Lies, damn lies and statistics ...

Weta Digital Ltd, a New Zealand-based animation company, renowned for the visual effects of “The Lord of the Rings” and “King Kong” movie blockbusters, implemented HP BL2x220c systems to create a high-performance platform that would increase processing density and reduce energy consumption. The system, consisting of four clusters equipped with 156 BL2x220c server blades each, ranks 219 through 222 on the current TOP500 list.

Source: http://finchannel.com/index.php?option=com_content&task=view&id=15203&Itemid=10

Wow ...

Greening the Data Centre (Anand Akela)

• Integrity Power Calculator uses your hardware actual configuration (eg. amount of RAM, speed of CPUs etc) to estimate power consumption.
• Once you've measured the actual power you can use Power Regulator to cap power usage to measured peak consumption or even measured average consumption.
• I find it ironic that the speaker says you can leverage all the saved power capacity to install more servers. I'm not sure how that's "greener"

More links:

• http://www.hp.com/go/integritythermallogic/
• http://www.hp.com/go/energyefficiency/

NFS Performance Comparison (Dave Olker)

• Filebench benchmark tool which allows customisation of the type of workload
• Performance comparisons for NFS + Kerberos: krb5 isn't too expensive but krb5i and krb5p are hugely expensive in theory ... however in the real world you typically get bottlenecked on disk
• TCP Segmentation Offload means that the OS sends 32k chunks of data to the NIC and then NIC itself breaks things up into MTU sized packets.
• NFSv3 + Kerberos will also replace the NFS RPC mechanism and thus work around the 16 group limitation. Sweet!
• They are seeing lots of customers using NFSv4 on HPUX, main driver is that it's easier to firewall NFSv4 due to its simpler layout of ports. Still not much traction with Linux that they are aware of.

Collectl: A Single Tool for all Your Linux Performance Monitoring Needs (Mark Seger)

• Homepage is at: http://collectl.sourceforge.net/
• Main features are efficient use of screen real estate, can monitor on sub-second intervals, interactive or record for future playback, has access to a wide range of data sources.
• It can output GNUplot formatted data for easy graphing (good for Excel as well)
• As of kernel 2.6.20 you can enable (CONFIG_TASKSTATS) real time per-process IO statistics (look in /proc/<pid>/io)
• CMU (Cluster Monitoring Utility) has an interesting way of displaying graphs for data about a whole cluster.
• colplot is a web front end for building GNUplot graphs from collectl data (it is not open source but can be made available to HP customers)

Mondo Rescue: A GPL Disaster Recovery Solution for Linux (Bruno Comec)

• Home page is at: http://trac.mondorescue.org/
• Creates a bootable ISO image (from a compressed afio stream) of the running Linux server including the currently running kernel and modules.
• Can save disk structure and Proliant hardware specific information.
• Can create an image from a live system.
• Can change filesystem type, layout or software RAID configuration when you restore (to Proliant hardware).
• PXE support means that you can dump your images to a central location and then serve them out to servers via network boot.
• List of HP supported open source projects: http://opensource.hp.com/opensource_projects.html

Built-in Linux Management and Deployment Tools for Proliant (Jonathan Anderson)

• SSST allows you to PXE boot, configure your hardware based on a template, configure your RAID, configure the iLO and even update all firmware.
• PSP (Proliant Support Package) is a bundled set of drivers, firmware and monitoring agents which have all been tested to work together.
• SSST is a cut down version of SLES9, the initrd.img is updated with all Proliant drivers. Handy!
• SSST will allow you to use a USB drive instead of PXE for the boot environment.
• Nice idea to setup PXE boot option to harvest the settings of a "gold server" and write it to a network location
• Utility dmidecode cross vendor tool for scanning the BIOS to determine hardware configuration, especially useful because every vendor has different tools for writing to BIOS.
• CONREP captures BIOS configuration to an XML file, allows the user to change it, and then write the new configuration back to the BIOS. TIP: edit the XML file to only contain the differences from default, this vastly speeds up the process of writing the configuration back to the hardware.
• CPQACUXE - allows you to pre-configure the RAID configuration
• HPONCFG - allows initial configuration of RILOE II, iLO and iLO2 from the host OS, without having to reboot
• CPQLOCFG or locfg.pl are intended for network administration of iLO's.
• Remote (Virtual) Serial Console can be accessed by telnet or SSH and provides access to POST, OS boot loader and OS itself (each has to be individually configured). Supports loading keys for passwordless SSH via the web GUI or the mxagentconfig utility.
• RIBCL - scripting language for pushing out changes to iLOs (and OAs?)
• Solution Demo Portal for online demonstrations of HP solutions
• Smart Components for updating ROM Flash locally, uses CPXXXXXX.scexe tools, requires the hpasm daemon to be running
• Example Perl scripts for remote managing iLOs http://www.hp.com/servers/lights-out (in Best Practices section), based around the locfg.pl command

• Configure Grub (or Lilo) to dump it's data out the virtual serial console:

  serial -unit=0 =speed=115200
  terminal -timeout=10 serial console
  ... <snip> ...
  kernel /vmlinuz-2.4.18-4smp ro root=/dev/sda9 console=tty= console=ttyS0,115200
  
  

• From the OA you can get a console window on any of the attached iLOs:

  CONNECT SERVER {SERIAL} <bay number>
  
  

• HPONCFG can connect to the OA and push out HPONCFG commands to iLOs. Interestingly HPONCFG can also download a RIBCL script from a URL (remember that the RIBCL command has the user/pass required for making changes to the iLO so security is important).

  HPONCFG [ ALL | <bay number> ] <from_url>
  
  

• Insight Online Diagnostics will allow you to diff a machine's current hardware and software configuration to a previous state or the state of another machine (requires hpasm and hpsmh).

• If you don't want to install all the HP driver crap, the two most useful ones are the health agent (hpasm) and the iLO interface driver (hpilo, was hprsm). Both hpasm and hpilo run in user space.
• OAs and iLOs can be configured to authenticate users via LDAP (and require group membership), it's advisable to keep a local administration user.

Laptop Management and Security in the Classroom (Andy Avery)

• Homepage is at: http://www.absolute.com/
• High end enterprise product is Computrace, consumer application is LoJack (I wondered what had happened to the source code)
• Installs a hidden daemon which checks in with an online server to see if it's been reported as stolen by the owner. If it has been, it gathers data and reports back.
• Runs on Mac and Windows and they have plans for Linux
• On Windows they have worked with all the major hardware vendors (HP, Dell etc) to get a shim put in the BIOS. The shim is deactivated by default but can be enabled by an administrator. Once it's enabled it installs the software. Basically it's a BIOS and boot sector resident virus.
• They say that roughly 70% of all stolen laptop's show up on the internet within 30 days. Of those laptops which show up they recover three out of four of them.
• They offer a guarantee of recovery, if you haven't got your laptop back within 60 days they'll pay up to $1000 to you.
• They also use their infrastructure to provide asset tracking for compliance and licensing.

All and all a pretty neat product, I'm off to get my free copy from their booth.

Getting Started with IPv6 (Mark Hollinger)

• Host queries will return both v4 and v6 records:

  overkill(shand)$ host ipv6.hp.com
  ipv6.hp.com has address 156.152.32.70
  ipv6.hp.com has IPv6 address 2620::a00:f101:156:152:32:70
  
  

• Term "subnet" has been replaced with the term "link" or "prefix"

• v6 addresses have a scope and a lifetime
• v6 doesn't have broadcast addresses, use multicast instead
• Made up of two parts, the prefix plus and interface id, so the routing information is separated from who you are
• Link local addressees start with fe80, they are automatically configured for each interface and only usable on local network
• A partial address ending in "::" means fill remaining address space with 0's

• Top level network prefixes and conventions which it's useful to know:

  ::/128 unspecified address, all zeros
  ::1/128 loopback address
  2001:db8::/32 is for documentation only
  2002::/16 6to4 addresses
  fc00::/8 globally unique local addresses
  fd00::/8 ula RFC4193 (equiv to RFC1918)
  ff00::/8 multicast
  
  

• Neighbour discovery is roughly equivalent to arp

• Stateless address auto-configuration means the host gets it's own IP based on an advertised router prefix
• Stateful address configuration means you use DHCPv6
• Right hand part of address used to be based on MAC address, this was great except had significant security concerns, fixed by using random number instead of MAC (RFC3041) vista uses this even for link local addresses (can be disabled)
• DHCPv6 can generate the RFC3041 randomised IP addresses for clients
• v6 allows you to have multiple IP's interfaces and you can regenerate RFC3041 addresses as often as you like. so you can regenerate IP's every 30 minutes and even have different browser windows (or applications) bound to different IP's. You could even have one IP for your work personality and another for your personal activity.
• Dynamic DNS exists for v6 and is increasingly important to deal with the length of v6 addresses
• You can put v6 addresses into URLs by using square brackets eg, http://[2620::...]/
• Google works on http://ipv6.google.com/

Implementing and Managing Linux and Solaris on Proliant (Marc Semadeni)

• HP is working hard to build a strong set of Linux management tools.
• Most people running Solaris (as opposed to Linux) are using it to host bespoke applications
• SSST (SmartStart Scripting Toolkit) mass deployment tools including firmware updates and configuration (RAID, HBA, iLO). Looks like it integrates with Kickstart on Linux.
• SIM (Systems Insight Manager) ... looks like OA interface for all HP stuff?
• ICE (Insight Control Environment) for Linux - role based access, bare metal discovery and deployment, power contorl, monitoring for alerts, history and reporting/graphing
• ServiceGuard - HA system for Oracle and SAP on both Solaris and Linux. Also supports Apache, MySQL, NFS, PostgreSQL, Samba, Sendmail and Tomcat. Multi server and multi data centre failover.

Networking Technology Roadmap

• SFP+ is a new lower power/cost version of an XFP (backed by Cisco but standards based?)
• New POE standard 802.3at can deliver up to 24W over cat5e
• WiMax has personal (802.16) and mobile versions (802.20)
• 802.11p is a new protocol to allow cars to talk to each other (eg. so one car can tell a car behind it that it's slammed on the brakes) cute.
• Evolution of GSM is LTE ... combined voice/data upto 100Mbps (down) and 50Mbps (up) at up to 15kph and up to 120kps at bullet train speeds.
• Power requirements are the main hold back on 10G-BASE-T
• 802.3az is a new low power spec for ethernet, it will use 2-3W less per ethernet port (yikes!)
• There's a new PCI spec for allowing virtualised hosts to talk directly to the hardware (scary ...)
• New term, wireless RAN for "Rolling Area Network"
• Information at http://www.hp.com/go/HPCpodcasts

Introduction to SELinux

• Utilities mostly use "-Z" switch for displaying SELinux info. (eg. "ls -Z", "ps -Z" or "id -Z")
• Manage security contexts with "chcon" and "restorecon"
• Commands "getenforce" and "setenforce" enable/disable SELinux
• RHEL SELinux supports three modes, enforcing (require SELinux to be configured correctly for the app), permissive (only warns instead of enforces) and disabled.
• "audit2allow" will let you create a security policy from the log file
• Dan Walsh is Red Hat SELinux project manager - http://danwalsh.livejournal.com/
• "SELinux by Example" by Frank Mayer is a great resource
• "SELinux: A New Approach to Secure Systems" white paper by Chris Runge
• Policy data is stored in the file inode and so it requires filesystem support.
• It can support files over NFS but only with a SELinux aware NFS server.

Hands On Troubleshooting and Security at the Packet Level (Laura Chappell)

• Wireshark University has lots of training videos and documentation.
• Pilot is an awesome looking new graphing front end for Wireshark (looks like it's Windows only, pah.)
• IronKey - USB thumb drive with built in encryption. After 10th password attempt it shreds the key. Recommended for carrying around unsanitised network traces.
• Lots of sample traces on the Laura's Lab Kit DVD, they show a wide range of interesting and broken behaviour. In the "LLK9 trace files" directory look for "tracefilelibrary2008.pdf" file which describes the traces and offers suggestions for learning from them.
• Apparently the latest Wireshark doesn't support plugins any more, additional functionality has to be compiled into the core (WTF?)
• CDP (Cisco Discovery Protocol) packets are multicast traffic, makes sense but I didn't realise that
• Multicast traffic always starts with 224-239.x.x.x
• IGMP is how hosts subscribe to multicast channels, allows switches to only forward traffic to ports which are interested
• Switched networks only show four types of traffic:
  • Broadcast
  • Multicast (if forwarded)
  • Traffic to/from your MAC address
  • Traffic to an unknown MAC address (hosts should always announce themselves when they boot up so they are typically in the switches arp table, this is a sign of weirdness)
• Switches forward packets to unknown MAC addresses to all ports in an attempt to find the unknown device. Switches won't quench packets to unknown MACs (really?)
• Macof is a network flooding tool (sends flood MAC addresses to an unknown MAC destination, eg. 88:88:88:88:88)
• Recommended debugging technique is plugging the users computer into a hub which you can share. Hubs are getting hard to find, most are switches these days.
• CACE is about to release a cheap gigabit network tap for capturing traffic from remote locations
• Turbocap - is a gig capable tap for a host (can forward packages to Pilot or Wireshark)
• Keep trace files under 20mb or Wireshark gets grumpy
• Most cards these days are in promiscuous mode by default (WTF?)
• Apparently Microsoft is internally getting rid of all network firewalls and third party firewall / anti-virus products because they are built into Vista's kernel. (WTF happened to defence in depth?)
• Wireshark colours things red as a warning that something is wrong. You can see what it is warning you about in the middle window underneath "Frame 1" in the "Colouring rules" tags.
• New tool called Wirebrush, makes it easy to sanitise traces (change's IPs to 10.x and changes passwords etc).
• Going to "View - Time Display Format - Seconds Since Previous Displayed Packet" ... really nice way of tracking latency in packets.
• Refusals which are normal (eg. trying to connect to a host/port on which there is nothing listening):
  • TCP port 80 - should get single RST packet
  • UDP port 53 - should get an ICMP destination/port unreachable
• Good list of protocol stuff at IANA: http://www.iana.org/protocols/
• A looped packet will have the same IP "Identification" header, a host which is spewing traffic will generate different "Identification" headers for each packet.
• Information in []'s is information from Wireshark, not from the actual trace.
• TCP Dup ACK is what you see when a packet is lost, it's the request for a retransmission. On high latency links it is normal to see many Dup ACKs before the server has a chance to reply.
• [Fast retransmission] is a misnomer, all it really means is that the retransmission happened within 20ms of the server response. It's going away because it implies a server centric POV and basically isn't useful.
• "Analyse - Expert Info Composite" has lots of cool summary information. Some of the decisions about what's an error/warning/note is a little wacky. They are trying to fix this now.
• By default in XP TCP window scaling is turned off. Turn it on for much improved download speeds (automatically resizes the window).
• TCP sequence numbers only increment when data is transmitted (eg. LEN>0), except for the first two and last two packets where there is a phantom byte and the sequence number increases even though no data was exchanged.
• There is a race condition in TCP, the final ACK packet doesn't have a phantom packet so the seq number doesn't get incremented. This means that if that final ACK gets lost on the way to the server, when the server sends a Dup ACK the client thinks it wants the first data packet (which has the same seq #) rather then the ACK. But the server won't accept data until it receives the ACK and the three way handshake is complete. SEE tcp-handshake-problem.pcap one the CD.
• ICMP type 13, 15 & 17 are very rarely used and are used for OS fingerprinting. Configure hosts to disable those types of ICMP to hinder fingerprinting.
• If you block all ICMP do not block ICMP type 3/code 4 (datagram too big) ... you'll end up with MTU issues.
• Hijack-this (www.spywarewarrior.com) is a good anti-spyware suite.
• ICMP redirects are unauthenticated and unpaired. Anybody can send an ICMP redirect to any other host and it will be honoured. Ettercap and Cain & Able are tools for doing this.
• Fragmentation happens at the IP level from MTU size mismatches. However the application can also segment the TCP traffic up into small pieces (eg. one byte per packet) in order to bypass firewall/IDS filters.

Good sites:

• http://www.remote-exploit.org/
• http://www.oxid.it/
• http://ettercap.sourceforge.net/
• http://www.usdoj.gov/criminal/cybercrime/

Since Baa Camp there's been a vast amount of interest in building a free wireless network in Wellington. It's awesome to see this interest take hold even if it does feel like a timewarp back to 2001 in Portland.

At the same time I'm hearing that Personal Telco is continuing to kick ass in Portland as MetroFi drops the ball, rock on guys.

Portland, Ore., considers its options with MetroFi's stalled network: The city of Portland alerted MetroFi in February that it considers the company "in default of contract," according to the (Portland) Oregonian. MetroFi told the paper that his firm won't be finishing the network without "financial support form the city and left open the possibility MetroFi will shut off the entire system." CEO Chuck Haas also seems to have sworn off ad-supported Wi-Fi, something the company switched to years ago, deciding there's truly not enough revenue there to turn a profit. Local group Personal Telco may move into a more leading role, given their steady work while MetroFi fiddled with their business model.

The Oregonian's blog cites some items from the 6 Feb. 2008 letter sent by Portland to MetroFi, noting a lack of ongoing communication and maintenance, as well as a failure to provide information about its advertising partner MSN's privacy practices.

Source: http://wifinetnews.com/archives/008284.html