f212ec4b7b
This patch adds a new configuration option, which adds support for a new
early_param which gets checked in arch/x86/kernel/setup_{32,64}.c:setup_arch()
to decide wether OHCI-1394 FireWire controllers should be initialized and
enabled for physical DMA access to allow remote debugging of early problems
like issues ACPI or other subsystems which are executed very early.
If the config option is not enabled, no code is changed, and if the boot
paramenter is not given, no new code is executed, and independent of that,
all new code is freed after boot, so the config option can be even enabled
in standard, non-debug kernels.
With specialized tools, it is then possible to get debugging information
from machines which have no serial ports (notebooks) such as the printk
buffer contents, or any data which can be referenced from global pointers,
if it is stored below the 4GB limit and even memory dumps of of the physical
RAM region below the 4GB limit can be taken without any cooperation from the
CPU of the host, so the machine can be crashed early, it does not matter.
In the extreme, even kernel debuggers can be accessed in this way. I wrote
a small kgdb module and an accompanying gdb stub for FireWire which allows
to gdb to talk to kgdb using remote remory reads and writes over FireWire.
An version of the gdb stub fore FireWire is able to read all global data
from a system which is running a a normal kernel without any kernel debugger,
without any interruption or support of the system's CPU. That way, e.g. the
task struct and so on can be read and even manipulated when the physical DMA
access is granted.
A HOWTO is included in this patch, in Documentation/debugging-via-ohci1394.txt
and I've put a copy online at
ftp://ftp.suse.de/private/bk/firewire/docs/debugging-via-ohci1394.txt
It also has links to all the tools which are available to make use of it
another copy of it is online at:
ftp://ftp.suse.de/private/bk/firewire/kernel/ohci1394_dma_early-v2.diff
Signed-Off-By: Bernhard Kaindl <bk@suse.de>
Tested-By: Thomas Renninger <trenn@suse.de>
Signed-off-by: Ingo Molnar <mingo@elte.hu>
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
179 lines
7.2 KiB
Text
179 lines
7.2 KiB
Text
|
|
Using physical DMA provided by OHCI-1394 FireWire controllers for debugging
|
|
---------------------------------------------------------------------------
|
|
|
|
Introduction
|
|
------------
|
|
|
|
Basically all FireWire controllers which are in use today are compliant
|
|
to the OHCI-1394 specification which defines the controller to be a PCI
|
|
bus master which uses DMA to offload data transfers from the CPU and has
|
|
a "Physical Response Unit" which executes specific requests by employing
|
|
PCI-Bus master DMA after applying filters defined by the OHCI-1394 driver.
|
|
|
|
Once properly configured, remote machines can send these requests to
|
|
ask the OHCI-1394 controller to perform read and write requests on
|
|
physical system memory and, for read requests, send the result of
|
|
the physical memory read back to the requester.
|
|
|
|
With that, it is possible to debug issues by reading interesting memory
|
|
locations such as buffers like the printk buffer or the process table.
|
|
|
|
Retrieving a full system memory dump is also possible over the FireWire,
|
|
using data transfer rates in the order of 10MB/s or more.
|
|
|
|
Memory access is currently limited to the low 4G of physical address
|
|
space which can be a problem on IA64 machines where memory is located
|
|
mostly above that limit, but it is rarely a problem on more common
|
|
hardware such as hardware based on x86, x86-64 and PowerPC.
|
|
|
|
Together with a early initialization of the OHCI-1394 controller for debugging,
|
|
this facility proved most useful for examining long debugs logs in the printk
|
|
buffer on to debug early boot problems in areas like ACPI where the system
|
|
fails to boot and other means for debugging (serial port) are either not
|
|
available (notebooks) or too slow for extensive debug information (like ACPI).
|
|
|
|
Drivers
|
|
-------
|
|
|
|
The OHCI-1394 drivers in drivers/firewire and drivers/ieee1394 initialize
|
|
the OHCI-1394 controllers to a working state and can be used to enable
|
|
physical DMA. By default you only have to load the driver, and physical
|
|
DMA access will be granted to all remote nodes, but it can be turned off
|
|
when using the ohci1394 driver.
|
|
|
|
Because these drivers depend on the PCI enumeration to be completed, an
|
|
initialization routine which can runs pretty early (long before console_init(),
|
|
which makes the printk buffer appear on the console can be called) was written.
|
|
|
|
To activate it, enable CONFIG_PROVIDE_OHCI1394_DMA_INIT (Kernel hacking menu:
|
|
Provide code for enabling DMA over FireWire early on boot) and pass the
|
|
parameter "ohci1394_dma=early" to the recompiled kernel on boot.
|
|
|
|
Tools
|
|
-----
|
|
|
|
firescope - Originally developed by Benjamin Herrenschmidt, Andi Kleen ported
|
|
it from PowerPC to x86 and x86_64 and added functionality, firescope can now
|
|
be used to view the printk buffer of a remote machine, even with live update.
|
|
|
|
Bernhard Kaindl enhanced firescope to support accessing 64-bit machines
|
|
from 32-bit firescope and vice versa:
|
|
- ftp://ftp.suse.de/private/bk/firewire/tools/firescope-0.2.2.tar.bz2
|
|
|
|
and he implemented fast system dump (alpha version - read README.txt):
|
|
- ftp://ftp.suse.de/private/bk/firewire/tools/firedump-0.1.tar.bz2
|
|
|
|
There is also a gdb proxy for firewire which allows to use gdb to access
|
|
data which can be referenced from symbols found by gdb in vmlinux:
|
|
- ftp://ftp.suse.de/private/bk/firewire/tools/fireproxy-0.33.tar.bz2
|
|
|
|
The latest version of this gdb proxy (fireproxy-0.34) can communicate (not
|
|
yet stable) with kgdb over an memory-based communication module (kgdbom).
|
|
|
|
Getting Started
|
|
---------------
|
|
|
|
The OHCI-1394 specification regulates that the OHCI-1394 controller must
|
|
disable all physical DMA on each bus reset.
|
|
|
|
This means that if you want to debug an issue in a system state where
|
|
interrupts are disabled and where no polling of the OHCI-1394 controller
|
|
for bus resets takes place, you have to establish any FireWire cable
|
|
connections and fully initialize all FireWire hardware __before__ the
|
|
system enters such state.
|
|
|
|
Step-by-step instructions for using firescope with early OHCI initialization:
|
|
|
|
1) Verify that your hardware is supported:
|
|
|
|
Load the ohci1394 or the fw-ohci module and check your kernel logs.
|
|
You should see a line similar to
|
|
|
|
ohci1394: fw-host0: OHCI-1394 1.1 (PCI): IRQ=[18] MMIO=[fe9ff800-fe9fffff]
|
|
... Max Packet=[2048] IR/IT contexts=[4/8]
|
|
|
|
when loading the driver. If you have no supported controller, many PCI,
|
|
CardBus and even some Express cards which are fully compliant to OHCI-1394
|
|
specification are available. If it requires no driver for Windows operating
|
|
systems, it most likely is. Only specialized shops have cards which are not
|
|
compliant, they are based on TI PCILynx chips and require drivers for Win-
|
|
dows operating systems.
|
|
|
|
2) Establish a working FireWire cable connection:
|
|
|
|
Any FireWire cable, as long at it provides electrically and mechanically
|
|
stable connection and has matching connectors (there are small 4-pin and
|
|
large 6-pin FireWire ports) will do.
|
|
|
|
If an driver is running on both machines you should see a line like
|
|
|
|
ieee1394: Node added: ID:BUS[0-01:1023] GUID[0090270001b84bba]
|
|
|
|
on both machines in the kernel log when the cable is plugged in
|
|
and connects the two machines.
|
|
|
|
3) Test physical DMA using firescope:
|
|
|
|
On the debug host,
|
|
- load the raw1394 module,
|
|
- make sure that /dev/raw1394 is accessible,
|
|
then start firescope:
|
|
|
|
$ firescope
|
|
Port 0 (ohci1394) opened, 2 nodes detected
|
|
|
|
FireScope
|
|
---------
|
|
Target : <unspecified>
|
|
Gen : 1
|
|
[Ctrl-T] choose target
|
|
[Ctrl-H] this menu
|
|
[Ctrl-Q] quit
|
|
|
|
------> Press Ctrl-T now, the output should be similar to:
|
|
|
|
2 nodes available, local node is: 0
|
|
0: ffc0, uuid: 00000000 00000000 [LOCAL]
|
|
1: ffc1, uuid: 00279000 ba4bb801
|
|
|
|
Besides the [LOCAL] node, it must show another node without error message.
|
|
|
|
4) Prepare for debugging with early OHCI-1394 initialization:
|
|
|
|
4.1) Kernel compilation and installation on debug target
|
|
|
|
Compile the kernel to be debugged with CONFIG_PROVIDE_OHCI1394_DMA_INIT
|
|
(Kernel hacking: Provide code for enabling DMA over FireWire early on boot)
|
|
enabled and install it on the machine to be debugged (debug target).
|
|
|
|
4.2) Transfer the System.map of the debugged kernel to the debug host
|
|
|
|
Copy the System.map of the kernel be debugged to the debug host (the host
|
|
which is connected to the debugged machine over the FireWire cable).
|
|
|
|
5) Retrieving the printk buffer contents:
|
|
|
|
With the FireWire cable connected, the OHCI-1394 driver on the debugging
|
|
host loaded, reboot the debugged machine, booting the kernel which has
|
|
CONFIG_PROVIDE_OHCI1394_DMA_INIT enabled, with the option ohci1394_dma=early.
|
|
|
|
Then, on the debugging host, run firescope, for example by using -A:
|
|
|
|
firescope -A System.map-of-debug-target-kernel
|
|
|
|
Note: -A automatically attaches to the first non-local node. It only works
|
|
reliably if only connected two machines are connected using FireWire.
|
|
|
|
After having attached to the debug target, press Ctrl-D to view the
|
|
complete printk buffer or Ctrl-U to enter auto update mode and get an
|
|
updated live view of recent kernel messages logged on the debug target.
|
|
|
|
Call "firescope -h" to get more information on firescope's options.
|
|
|
|
Notes
|
|
-----
|
|
Documentation and specifications: ftp://ftp.suse.de/private/bk/firewire/docs
|
|
|
|
FireWire is a trademark of Apple Inc. - for more information please refer to:
|
|
http://en.wikipedia.org/wiki/FireWire
|