diff --git a/.gitattributes b/.gitattributes index 04b1ff2926..3b556a8b3a 100644 --- a/.gitattributes +++ b/.gitattributes @@ -3,53 +3,79 @@ # Explicitly declare text files you want to always be normalized and converted # to native line endings on checkout. -*.java text -*.gradle text -*.manifest text +*.bash text +*.c text +*.cc text +*.command text +*.cpp text +*.cspec text *.css text +*.gradle text +*.groovy text +*.h text +*.hh text *.htm text *.html text +*.java text +*.jj text *.js text *.json text *.jsp text *.jspf text *.jspx text +*.l text +*.lang text +*.ldefs text +*.manifest text +*.opinion text +*.props text *.properties text -*.tld text -*.txt text +*.proto text +*.pspec text +*.py text +*.rxg text +*.sh text +*.sla text *.tag text +*.tld text +*.tool text +*.trans text +*.txt text *.xml text -*.c text -*.h text -*.cpp text -*.hh text -*.cc text +*.y text # Declare files that will always have CRLF line endings on checkout. +*.bat text eol=crlf *.sln text eol=crlf *.vcproj text eol=crlf *.vcxproj text eol=crlf -*.bat text eol=crlf # Denote all files that are truly binary and should not be modified. -*.png binary -*.jpg binary +*.a binary +*.apk binary +*.bmp binary *.class binary *.dll binary +*.dmg binary *.ear binary -*.gif binary -*.ico binary -*.jar binary -*.jpeg binary -*.so binary -*.war binary -*.pdf binary *.exe binary -*.lib binary -*.sa binary +*.gdt binary +*.gif binary *.gz binary *.gzf binary -*.tgz binary +*.ico binary +*.ipsw binary +*.jar binary +*.jpeg binary +*.jpg binary +*.lib binary +*.o binary +*.obj binary +*.pdf binary +*.png binary +*.sa binary +*.so binary *.tar binary -*.sh binary - +*.tgz binary +*.war binary +*.zip binary diff --git a/.gitignore b/.gitignore index a2896ae8a3..33f03c2bd7 100644 --- a/.gitignore +++ b/.gitignore @@ -3,6 +3,7 @@ ghidra.repos.config # Misc files produced while executing application repositories/ +dependencies/ flatRepo/ Ghidra/.ghidraSvrKeys wrapper.log* @@ -84,4 +85,6 @@ gradle/wrapper gradlew gradlew.* - +# Ignore logs and core dumps +*.log +core.* diff --git a/DebuggerDevGuide.md b/DebuggerDevGuide.md new file mode 100644 index 0000000000..b3731a5d66 --- /dev/null +++ b/DebuggerDevGuide.md @@ -0,0 +1,107 @@ +# Debugger Developer's Guide + +## Catalog of Dependencies + +In addition to Ghidra's normal dependencies, you may want the following: + + * WinDbg for Windows x64 + * GDB 8.0 or later for Linux amd64/x86_64 + +The others (e.g., JNA) are handled by Gradle via Maven Central. + +## Architecture Overview + +There are several Eclipse projects each fitting into a larger architectural picture. +These all currently reside in the `Ghidra/Debug` directory, but will likely be re-factored into the `Framework` and `Feature` directories later. +Each project is listed "bottom up" with a brief description and status. + + * ProposedUtils - a collection of utilities proposed to be moved to other respective projects + * AnnotationValidator - an experimental annotation processor for database access objects + * Framework-TraceModeling - a database schema and set of interfaces for storing machine state over time + * Framework-AsyncComm - a collection of utilities for asynchronous communication (packet formats and completable-future conveniences). + * Framework-Debugging - specifies interfaces for debugger models and provides implementation conveniences. + * Debugger - the collection of Ghidra plugins and services comprising the Debugger UI. + * Debugger-agent-dbgeng - the connector for WinDbg (via dbgeng.dll) on Windows x64. + * Debugger-agent-dbgmodel - an experimental connector for WinDbg Preview (with TTD, via dbgmodel.dll) on Windows x64. + * Debugger-agent-dbgmodel-traceloader - an experimental "importer" for WinDbg trace files. + * Debugger-agent-gdb - the connector for GDB (8.0 or later recommended) on UNIX. + * Debugger-gadp - the connector for our custom wire protocol the Ghidra Asynchronous Debugging Protocol. + * Debugger-jpda - an in-development connector for Java and Dalvik debugging via JDI (i.e., JDWP). + * Debugger-sctl - a deprecated connector for the SCTL stub (cqctworld.org). + +The Trace Modeling schema records machine state and markup over time. +It rests on the same database as Programs, allowing trace recordings to be stored in a Ghidra project and shared via a server, if desired. +Trace "recording" is a de facto requirement for displaying information in Ghidra's UI. +However, only the machine state actually observed by the user (or perhaps a script) is recorded. +For most use cases, the Trace is small and ephemeral, serving only to mediate between the UI components and the target's model. +It supports many of the same markup (e.g., disassembly, data types) as Programs, in addition to tracking active threads, loaded modules, breakpoints, etc. + +Every model (or "adapter" or "connector" or "agent") implements the API specified in Framework-Debugging. As a general rule in Ghidra, no component is allowed to access a native API and reside in the same JVM as the Ghidra UI. +This allows us to contain crashes, preventing data loss. +To accommodate this requirement -- given that debugging native applications is almost certainly going to require access to native APIs -- we've developed the Ghidra Asynchronous Debugging Protocol. +This protocol is tightly coupled to Framework-Debugging, essentially exposing its methods via RMI. +The protocol is built using Google's Protobuf library, providing a potential path for agent implementations in alternative languages. +GADP provides both a server and a client implementation. +The server can accept any model which adheres to the specification and expose it via TCP; the client does the converse. +When a model is instantiated in this way, it is called an "agent," because it is executing in its own JVM. +The other connectors, which do not use native APIs, may reside in Ghidra's JVM and typically implement alternative wire protocols, e.g., JDWP and SCTL. +In both cases, the implementations inherit from the same interfaces. + +The Debugger services maintain a collection of active connections and inspect each model for potential targets. +When a target is found, the service inspects the target environment and attempts to find a suitable opinion. +Such an opinion, if found, instructs Ghidra how to map the objects, addresses, registers, etc. from the target namespace into Ghidra's. +The target is then handed to a Trace Recorder which begins collecting information needed to populate the UI, e.g., the program counter, stack pointer, and the bytes of memory they refer to. + +## Developing a new connector + +So Ghidra does not yet support your favorite debugger? +It is tempting, exciting, but also daunting to develop your own connector. +Please finish reading this guide, and look carefully at the ones we have so far, and perhaps ask to see if we are already developing one. +Of course, in time you might also search the internet to see if others are developing one. +There are quite a few caveats and gotchas, the most notable being that this interface is still in quite a bit of flux. +When things go wrong, it could be because of, without limitation: 1) a bug on your part, 2) a bug on our part, 3) a design flaw in the interfaces, or 4) a bug in the debugger/API your adapting. +We are still in the process of writing up this documentation. +In the meantime, we recommend using the GDB and dbgeng.dll agents as examples. + +You'll also need to provide launcher(s) so that Ghidra knows how to configure and start your connector. +Please provide launchers for your model in both configurations: as a connector in Ghidra's JVM, and as a GADP agent. +If your model requires native API access, you should only permit launching it as a GADP agent, unless you give ample warning in the launcher's description. +Look at the existing launchers for examples. +There are many model implementation requirements that cannot be expressed in Java interfaces. +Failing to adhere to those requirements may cause different behaviors with and without GADP. +Testing with GADP tends to reveal those implementation errors, but also obscures the source of client method calls behind network messages. + +## Adding a new platform + +If an existing connector exists for a suitable debugger on the desired platform, then adding it may be very simple. +For example, both the x86 and ARM platforms are supported by GDB, so even though we're currently focused on x86 support, we've provided the opinions needed for Ghidra to debug ARM platforms via GDB. +These opinions are kept in the "Debugger" project, not their respective "agent" projects. +We imagine there are a number of platforms that could be supported almost out of the box, except that we haven't written the necessary opinions, yet. +Take a look at the existing ones for examples. + +In general, to write a new opinion, you need to know: 1) What the platform is called (including variant names) by the debugger, 2) What the processor language is called by Ghidra, 3) If applicable, the mapping of target address spaces into Ghidra's address spaces, 4) If applicable, the mapping of target register names to those in Ghidra's processor language. +In most cases (3) and (4) are already implemented by default mappers, so you can use those same mappers in your opinion. +Once you have the opinion written, you can try debugging and recording a target. +If Ghidra finds your opinion applicable to that target, it will attempt to record, and then you can work out the kinds from there. +Again, we have a bit of documentation to do regarding common pitfalls. + +## Emulation + +It may be tempting to write a "connector" for emulation, but we recommend against it. +We are exploring the inclusion of emulation as an integral feature of the UI. +Namely for interpolation between machines states recorded in a trace, and extrapolation into future machine states. +In other words, a connector for emulation is likely to be deprecated by our future work. + +## Contributing + +Whether submitting help tickets and pull requests, please tag those related to the debugger with "Debugger" so that we can triage them more quickly. + +To set up your environment, in addition to the usual Gradle tasks, process the Protobuf specification for GADP: + +```bash +gradle generateProto +``` + +If you already have an environment set up in Eclipse, please re-run `gradle prepDev eclipse` and import the new projects. +The Protobuf plugin for Gradle does not seem to export the generated source directory to the Eclipse project. +To remedy this, add `build/generated/source/proto/main/java` to the build path, and configure it to output to `bin/main`. \ No newline at end of file diff --git a/DevGuide.md b/DevGuide.md index eceb003b58..991c09aa8a 100644 --- a/DevGuide.md +++ b/DevGuide.md @@ -21,6 +21,7 @@ - [Building Supporting Data](#building-supporting-data) * [Building Data Type Archives](#building-data-type-archives) * [Building FID Databases](#building-fid-databases) +- [Hacking on the Debugger](#hacking-on-the-debugger) ## Catalog of Dependencies @@ -33,7 +34,7 @@ You may not need all of these, depending on which portions you are building or d - https://adoptopenjdk.net/releases.html?variant=openjdk11&jvmVariant=hotspot - Amazon Corretto - https://docs.aws.amazon.com/corretto/latest/corretto-11-ug/downloads-list.html -* Gradle 5.0 or later - We use version 5.0, and tested with up to 5.6.3. +* Gradle 5.0 or later - We use version 5.0, and tested with up to 6.8.3. - https://gradle.org/next-steps/?version=5.0&format=bin * A C/C++ compiler - We use GCC on Linux, Xcode (Clang) on macOS, and Visual Studio (2017 or later) on Windows. - https://gcc.gnu.org/ @@ -61,9 +62,9 @@ You may not need all of these, depending on which portions you are building or d - https://sourceforge.net/projects/yajsw/files/yajsw/yajsw-stable-12.12/ * Eclipse PDE - Environment for developing the GhidraDev plugin. - https://www.eclipse.org/pde/ -* Eclipse CDT. We use version 8.6.0 - Build dependency for the GhidraDev plugin. +* Eclipse CDT. We build against version 8.6.0 - Build dependency for the GhidraDev plugin. - https://www.eclipse.org/cdt/ -* PyDev. We use version 6.3.1 - Build dependency for the GhidraDev plugin. +* PyDev. We build against version 6.3.1 - Build dependency for the GhidraDev plugin. - https://sourceforge.net/projects/pydev/files/pydev/ There are many, many others automatically downloaded by Gradle from Maven Central and Bintray JCenter when building and/or setting up the development environment. @@ -110,64 +111,67 @@ or manually by downloading the required dependencies. Choose one of the two fol The flat directory-style repository can be setup automatically by running a simple Gradle script. Navigate to `~/git/ghidra` and run the following: ``` -gradle --init-script gradle/support/fetchDependencies.gradle init +gradle -I gradle/support/fetchDependencies.gradle init ``` The Gradle task to be executed, in this case _init_, is unimportant. The point is to have Gradle execute -the `fetchDependencies.gradle` script. If it ran correctly you will have a new `~/git/ghidra/flatRepo/` -directory populated with the following jar files: - * AXMLPrinter2 - * csframework - * dex-ir-2.0 - * dex-reader-2.0 - * dex-reader-api-2.0 - * dex-tools-2.0 - * dex-translator-2.0 - * dex-writer-2.0 - * hfsx - * hfsx_dmglib - * iharder-base64 - -There will also be a new archive files at: - * ~/git/ghidra/Ghidra/Features/GhidraServer/build/`yajsw-stable-12.12.zip` - * ~/git/ghidra/GhidraBuild/EclipsePlugins/GhidraDev/GhidraDevPlugin/build/`PyDev 6.3.1.zip` - * ~/git/ghidra/GhidraBuild/EclipsePlugins/GhidraDev/GhidraDevPlugin/build/`cdt-8.6.0.zip` +the `fetchDependencies.gradle` script. If it ran correctly you will have a new `~/git/ghidra/dependencies/` +directory populated with the following files: + * flatRepo/AXMLPrinter2.jar + * flatRepo/csframework.jar + * flatRepo/dex-ir-2.0.jar + * flatRepo/dex-reader-2.0.jar + * flatRepo/dex-reader-api-2.0.jar + * flatRepo/dex-tools-2.0.jar + * flatRepo/dex-translator-2.0.jar + * flatRepo/dex-writer-2.0.jar + * flatRepo/hfsx.jar + * flatRepo/hfsx_dmglib.jar + * flatRepo/iharder-base64.jar + * cdt-8.6.0.zip + * PyDev 6.3.1.zip + * yajsw-stable-12.12.zip + * fidb/*.fidb If you see these, congrats! Skip to [building](#building-ghidra) or [developing](#developing-ghidra). If not, continue with manual download instructions below... ### Manual Download Instructions -Create the `~/git/ghidra/flatRepo/` directory to hold the manually-downloaded dependencies: +Create the `~/git/ghidra/dependencies/` directory and required subdirectories to hold the manually-downloaded dependencies: ```bash -mkdir ~/git/ghidra/flatRepo +mkdir ~/git/ghidra/dependencies +mkdir ~/git/ghidra/dependencies/flatRepo +mkdir ~/git/ghidra/dependencies/fidb +mkdir ~/git/ghidra/dependencies/GhidraServer +mkdir ~/git/ghidra/dependencies/GhidraDev ``` #### Get Dependencies for FileFormats: Download `dex-tools-2.0.zip` from the dex2jar project's releases page on GitHub. -Unpack the `dex-*.jar` files from the `lib` directory to `~/git/ghidra/flatRepo`: +Unpack the `dex-*.jar` files from the `lib` directory to `~/git/ghidra/dependencies/flatRepo`: ```bash cd ~/Downloads # Or wherever curl -OL https://github.com/pxb1988/dex2jar/releases/download/2.0/dex-tools-2.0.zip unzip dex-tools-2.0.zip -cp dex2jar-2.0/lib/dex-*.jar ~/git/ghidra/flatRepo/ +cp dex2jar-2.0/lib/dex-*.jar ~/git/ghidra/dependencies/flatRepo/ ``` Download `AXMLPrinter2.jar` from the "android4me" archive on code.google.com. -Place it in `~/git/ghidra/flatRepo`: +Place it in `~/git/ghidra/dependencies/flatRepo`: ```bash -cd ~/git/ghidra/flatRepo +cd ~/git/ghidra/dependencies/flatRepo curl -OL https://storage.googleapis.com/google-code-archive-downloads/v2/code.google.com/android4me/AXMLPrinter2.jar ``` #### Get Dependencies for DMG: Download `hfsexplorer-0_21-bin.zip` from www.catacombae.org. -Unpack the `lib` directory to `~/git/ghidra/flatRepo`: +Unpack the `lib` directory to `~/git/ghidra/dependencies/flatRepo`: ```bash cd ~/Downloads # Or wherever @@ -176,35 +180,33 @@ mkdir hfsx cd hfsx unzip ../hfsexplorer-0_21-bin.zip cd lib -cp csframework.jar hfsx_dmglib.jar hfsx.jar iharder-base64.jar ~/git/ghidra/flatRepo/ +cp csframework.jar hfsx_dmglib.jar hfsx.jar iharder-base64.jar ~/git/ghidra/dependencies/flatRepo/ ``` #### Get Dependencies for GhidraServer Building the GhidraServer requires "Yet another Java service wrapper" (yajsw) version 12.12. Download `yajsw-stable-12.12.zip` from their project on www.sourceforge.net, and place it in: -`~/git/ghidra/Ghidra/Features/GhidraServer/build`: +`~/git/ghidra/dependencies/GhidraServer/`: ```bash cd ~/Downloads # Or wherever curl -OL https://sourceforge.net/projects/yajsw/files/yajsw/yajsw-stable-12.12/yajsw-stable-12.12.zip -mkdir -p ~/git/ghidra/Ghidra/Features/GhidraServer/build/ -cp ~/Downloads/yajsw-stable-12.12.zip ~/git/ghidra/Ghidra/Features/GhidraServer/build/ +cp ~/Downloads/yajsw-stable-12.12.zip ~/git/ghidra/dependencies/GhidraServer/ ``` #### Get Dependencies for GhidraDev Building the GhidraDev plugin for Eclipse requires the CDT and PyDev plugins for Eclipse. Download `cdt-8.6.0.zip` from The Eclipse Foundation, and place it in: -`~/git/ghidra/GhidraBuild/EclipsePlugins/GhidraDev/GhidraDevPlugin/build/`: +`~/git/ghidra/dependencies/GhidraDev/`: ```bash cd ~/Downloads # Or wherever curl -OL 'https://archive.eclipse.org/tools/cdt/releases/8.6/cdt-8.6.0.zip' curl -o 'cdt-8.6.0.zip.sha512' -L --retry 3 'https://www.eclipse.org/downloads/sums.php?type=sha512&file=/tools/cdt/releases/8.6/cdt-8.6.0.zip' shasum -a 512 -c 'cdt-8.6.0.zip.sha512' -mkdir -p ~/git/ghidra/GhidraBuild/EclipsePlugins/GhidraDev/GhidraDevPlugin/build/ -cp ~/Downloads/cdt-8.6.0.zip ~/git/ghidra/GhidraBuild/EclipsePlugins/GhidraDev/GhidraDevPlugin/build/ +cp ~/Downloads/cdt-8.6.0.zip ~/git/ghidra/dependencies/GhidraDev/ ``` Download `PyDev 6.3.1.zip` from www.pydev.org, and place it in the same directory: @@ -212,7 +214,27 @@ Download `PyDev 6.3.1.zip` from www.pydev.org, and place it in the same director ```bash cd ~/Downloads # Or wherever curl -L -o 'PyDev 6.3.1.zip' https://sourceforge.net/projects/pydev/files/pydev/PyDev%206.3.1/PyDev%206.3.1.zip -cp ~/Downloads/'PyDev 6.3.1.zip' ~/git/ghidra/GhidraBuild/EclipsePlugins/GhidraDev/GhidraDevPlugin/build/ +cp ~/Downloads/'PyDev 6.3.1.zip' ~/git/ghidra/dependencies/GhidraDev/ +``` + +#### Get Ghidra Function ID datasets + +Download the Ghidra Function ID dataset files from the `ghidra-data` GitHub repository and place them +in `~/git/ghidra/dependencies/fidb`: + +```bash +cd ~/Downloads # Or wherever +curl -L -o 'vs2012_x64.fidb' https://github.com/NationalSecurityAgency/ghidra-data/raw/master/FunctionID/vs2012_x64.fidb +curl -L -o 'vs2012_x86.fidb' https://github.com/NationalSecurityAgency/ghidra-data/raw/master/FunctionID/vs2012_x86.fidb +curl -L -o 'vs2015_x64.fidb' https://github.com/NationalSecurityAgency/ghidra-data/raw/master/FunctionID/vs2015_x64.fidb +curl -L -o 'vs2015_x86.fidb' https://github.com/NationalSecurityAgency/ghidra-data/raw/master/FunctionID/vs2015_x86.fidb +curl -L -o 'vs2017_x64.fidb' https://github.com/NationalSecurityAgency/ghidra-data/raw/master/FunctionID/vs2017_x64.fidb +curl -L -o 'vs2017_x86.fidb' https://github.com/NationalSecurityAgency/ghidra-data/raw/master/FunctionID/vs2017_x86.fidb +curl -L -o 'vs2019_x64.fidb' https://github.com/NationalSecurityAgency/ghidra-data/raw/master/FunctionID/vs2019_x64.fidb +curl -L -o 'vs2019_x86.fidb' https://github.com/NationalSecurityAgency/ghidra-data/raw/master/FunctionID/vs2019_x86.fidb +curl -L -o 'vsOlder_x64.fidb' https://github.com/NationalSecurityAgency/ghidra-data/raw/master/FunctionID/vsOlder_x64.fidb +curl -L -o 'vsOlder_x86.fidb' https://github.com/NationalSecurityAgency/ghidra-data/raw/master/FunctionID/vsOlder_x86.fidb +cp ~/Downloads/*.fidb ~/git/ghidra/dependencies/fidb/ ``` ## Building Ghidra @@ -376,4 +398,11 @@ Fill out the options appropriately and click OK. If you'd like some details of our fine tuning, take a look at `~/git/ghidra/Ghidra/Features/FunctionID/data/building_fid.txt`. +### Hacking on the Debugger + +The Debugger consists of multiple modules comprising its own collection of utilities, frameworks, and features. +There is plenty of new ground to be broken. +Before getting too deep into it, please see our dedicated [Debugger Developer's Guide][DbgGuide]. + [ghidra-data]: https://github.com/NationalSecurityAgency/ghidra-data +[DbgGuide]: DebuggerDevGuide.md \ No newline at end of file diff --git a/GPL/GnuDisassembler/certification.manifest b/GPL/GnuDisassembler/certification.manifest index 5ef30394a2..1692c70354 100644 --- a/GPL/GnuDisassembler/certification.manifest +++ b/GPL/GnuDisassembler/certification.manifest @@ -1,7 +1,6 @@ ##VERSION: 2.0 ##MODULE IP: GPL 2 ##MODULE IP: Public Domain -.project||GHIDRA||||END| Module.manifest||Public Domain||||END| README.txt||Public Domain||||END| data/arm_test1.s||Public Domain||||END| diff --git a/Ghidra/Configurations/Public_Release/src/global/docs/ChangeHistory.html b/Ghidra/Configurations/Public_Release/src/global/docs/ChangeHistory.html index 98bad04149..21f8caaf81 100644 --- a/Ghidra/Configurations/Public_Release/src/global/docs/ChangeHistory.html +++ b/Ghidra/Configurations/Public_Release/src/global/docs/ChangeHistory.html @@ -6,6 +6,19 @@ +

Ghidra 9.2.4 Change History (April 2021)

+

Improvements

+ +
+

Bugs

+ +
+

Ghidra 9.2.3 Change History (March 2021)

Improvements