GP-4173 Complete transition to using --option params with bsim and bsim_ctl commands

This commit is contained in:
ghidra1
2024-01-03 17:29:45 -05:00
parent 0de3f53c11
commit 735cb1a5c3
12 changed files with 830 additions and 718 deletions
@@ -69,7 +69,7 @@ This is equivalent to:
Use a command-line like this to generate and commit signatures from a Ghidra Server
repository to the Elasticsearch database created above:
bsim generatesigs ghidra://1.2.3.4/repo bsim=elastic://1.2.3.4:9200/repo
bsim generatesigs ghidra://1.2.3.4/repo --bsim elastic://1.2.3.4:9200/repo
Within Ghidra's BSim client, enter the same URL into the database connection
panel in order to place queries to your Elasticsearch deployment. See the BSim
File diff suppressed because it is too large Load Diff
@@ -214,7 +214,7 @@
<TR>
<TD><CODE class="computeroutput">$(ROOT)/support/bsim_ctl start /path/to/datadir
port=8000</CODE></TD>
--port&nbsp;8000</CODE></TD>
</TR>
<TR>
@@ -236,7 +236,7 @@
be reused.</P>
<P>The <SPAN class="bold"><STRONG>start</STRONG></SPAN> command can take an optional
<SPAN class="bold"><STRONG>port=</STRONG></SPAN> parameter. This can be used to specify
<SPAN class="bold"><STRONG>--port</STRONG></SPAN> parameter. This can be used to specify
a non-standard port for the PostgreSQL server to listen on. In this case, any
subsequent reference to the BSim server, in the Ghidra client, or with the <SPAN class=
"command"><STRONG>bsim</STRONG></SPAN> command described below, must specify the port.
@@ -293,7 +293,7 @@
<DD>
<P><CODE class="computeroutput">bsim_ctl start /path/to/datadir
auth=trust</CODE></P>
--auth&nbsp;trust</CODE></P>
<P>This is currently the default. No authentication is performed and privilege
is granted based on the user name presented. Masquerading is possible.</P>
@@ -304,7 +304,7 @@
<DD>
<P><CODE class="computeroutput">bsim_ctl start /path/to/datadir
auth=password</CODE></P>
--auth&nbsp;password</CODE></P>
<P>Users are authenticated via password. A default password 'changeme' is
established when the new user is created. Passwords can be changed by the user
@@ -315,12 +315,12 @@
<DT><SPAN class="term"><SPAN class="bold"><STRONG>pki</STRONG></SPAN></SPAN></DT>
<DD>
<P><CODE class="computeroutput">bsim_ctl start /path/to/datadir auth=pki
ca=/path/to/rootcert</CODE></P>
<P><CODE class="computeroutput">bsim_ctl start /path/to/datadir --auth&nbsp;pki
--cafile&nbsp;"/path/to/rootcert"</CODE></P>
<P>Users are authenticated by PKI certificates. Upon initialization, the BSim
server must be provided (via the <SPAN class=
"command"><STRONG>ca=</STRONG></SPAN> option) a file containing the public keys
"command"><STRONG>--cafile</STRONG></SPAN> option) a file containing the public keys
for the certificate authorities used to issue user's certificates. The file
consists of the authoritative certificates in PEM format concatenated
together.</P>
@@ -338,7 +338,7 @@
<P>With PKI authentication enabled, at the time a new user role is established
with the server, the X.509 Distinguished Name, as bound to the user's
certificate, must be associated with the user name via the <SPAN class=
"command"><STRONG>dn=</STRONG></SPAN> option. See <A class="xref" href=
"command"><STRONG>--dn</STRONG></SPAN> option. See <A class="xref" href=
"#PostAddUser" title="Adding Users to the Database">&ldquo;Adding Users to the
Database&rdquo;</A>.</P>
</DD>
@@ -358,7 +358,7 @@
<TABLE border="0" summary="Simple list" class="simplelist">
<TR>
<TD><CODE class="computeroutput">$(ROOT)/support/bsim_ctl changeauth
/datadir/path auth=password</CODE></TD>
/datadir/path --auth&nbsp;password</CODE></TD>
</TR>
</TABLE>
</DIV>
@@ -401,7 +401,7 @@
<TR>
<TD><CODE class="computeroutput">$(ROOT)/support/bsim_ctl adduser <SPAN class=
"emphasis"><EM>username</EM></SPAN> dn="C=US,ST=MD,CN=Firstname User"</CODE></TD>
"emphasis"><EM>username</EM></SPAN> --dn&nbsp;"C=US,ST=MD,CN=Firstname User"</CODE></TD>
</TR>
</TABLE>
</DIV>
@@ -410,7 +410,7 @@
initially be set to 'changeme'. If PKI authentication has been set for the server, The
Distinguished Name, as bound to the new user's certificated must be provided when
issuing the <SPAN class="command"><STRONG>adduser</STRONG></SPAN> command, via the
<SPAN class="command"><STRONG>dn=</STRONG></SPAN> option. The Distinguished Name must
<SPAN class="command"><STRONG>--dn</STRONG></SPAN> option. The Distinguished Name must
be presented as a string containing a comma separated sequence of attribute/value pairs
that uniquely identifies a certificate. Currently, the Common Name (CN=) is the only
attribute inspected by the PostgreSQL server, so other attributes can be omitted.</P>
@@ -836,17 +836,17 @@ curl -k -u elastic:XXXXXX -X POST "https://localhost:9200/_security/user/ghidrau
<TABLE border="0" summary="Simple list" class="simplelist">
<TR>
<TD><CODE class="computeroutput">$(ROOT)/support/bsim setmetadata <SPAN class=
"emphasis"><EM>bsimURL</EM></SPAN> "name=BSim Database"</CODE></TD>
"emphasis"><EM>bsimURL</EM></SPAN> --name&nbsp;"BSim Database"</CODE></TD>
</TR>
<TR>
<TD><CODE class="computeroutput">$(ROOT)/support/bsim setmetadata <SPAN class=
"emphasis"><EM>bsimURL</EM></SPAN> "owner=Administrators"</CODE></TD>
"emphasis"><EM>bsimURL</EM></SPAN> --owner&nbsp;"Administrators"</CODE></TD>
</TR>
<TR>
<TD><CODE class="computeroutput">$(ROOT)/support/bsim setmetadata <SPAN class=
"emphasis"><EM>bsimURL</EM></SPAN> "description=Files of interest"</CODE></TD>
"emphasis"><EM>bsimURL</EM></SPAN> --description&nbsp;"Files of interest"</CODE></TD>
</TR>
</TABLE>
</DIV>
@@ -855,8 +855,8 @@ curl -k -u elastic:XXXXXX -X POST "https://localhost:9200/_security/user/ghidrau
changed at any time and do not otherwise affect the records contained in the database.
Multiple command-line parameters can be fed to <SPAN class="command"><STRONG>bsim
setmetadata</STRONG></SPAN> so long as each one starts with <SPAN class=
"bold"><STRONG>name=</STRONG></SPAN>, <SPAN class="bold"><STRONG>owner=</STRONG></SPAN>, or
<SPAN class="bold"><STRONG>description=</STRONG></SPAN> respectively. Quoting may be
"bold"><STRONG>--name</STRONG></SPAN>, <SPAN class="bold"><STRONG>--owner</STRONG></SPAN>, or
<SPAN class="bold"><STRONG>--description</STRONG></SPAN> respectively. Quoting of values may be
necessary to get some strings to be interpreted as a single command-line parameter.</P>
<DIV class="sect2">
@@ -112,20 +112,20 @@
"command"><STRONG>bsim generatesigs</STRONG></SPAN> command. Signatures may be written as
XML files to a local directory and/or committed directly to a specified BSim database. If
not immediately committing to a database and only storing the XML files an appropriate
database <EM>config=</EM> may be specified in lieu of a BSim database URL
(<EM>bsimURL</EM>) if database specific executable categories and function tags are not
utilized. Use of the <EM>config=</EM> option does not require a running BSim server.</P>
database configuration may be specified using the <EM>--config</EM> option in lieu of a BSim database URL
(--bsim <EM>&lt;bsimURL&gt;</EM>) if database specific executable categories and function tags are not
utilized. Use of the <EM>--config</EM> option does not require a running BSim server.</P>
<DIV class="informalexample">
<TABLE border="0" summary="Simple list" class="simplelist">
<TR>
<TD><CODE class="computeroutput">$(ROOT)/support/bsim generatesigs
&lt;ghidraURL&gt; &lt;/xmldirectory&gt; config=&lt;config_template&gt;
&lt;ghidraURL&gt; &lt;/xmldirectory&gt; --config&nbsp;&lt;config_template&gt;
[--overwrite]<BR>
$(ROOT)/support/bsim generatesigs &lt;ghidraURL&gt; &lt;/xmldirectory&gt;
bsim=&lt;bsimURL&gt; [--commit] [--overwrite]<BR>
--bsim&nbsp;&lt;bsimURL&gt; [--commit] [--overwrite]<BR>
$(ROOT)/support/bsim generatesigs &lt;ghidraURL&gt;
bsim=&lt;bsimURL&gt;</CODE></TD>
--bsim&nbsp;&lt;bsimURL&gt;</CODE></TD>
</TR>
</TABLE>
</DIV>
@@ -137,7 +137,7 @@
<TR>
<TD><CODE class="computeroutput">$(ROOT)/support/bsim generatesigs
ghidra://localhost/repo/folder /xmldirectory
bsim=postgresql://localhost/repo</CODE></TD>
--bsim&nbsp;postgresql://localhost/repo</CODE></TD>
</TR>
</TABLE>
</DIV>
@@ -148,7 +148,7 @@
<TABLE border="0" summary="Simple list" class="simplelist">
<TR>
<TD><CODE class="computeroutput">$(ROOT)/support/bsim generatesigs
ghidra://localhost/repo/folder /xmldirectory bsim=postgresql://localhost/repo
ghidra://localhost/repo/folder /xmldirectory --bsim&nbsp;postgresql://localhost/repo
--commit</CODE></TD>
</TR>
</TABLE>
@@ -176,7 +176,7 @@
the signature generation process, such as database specific executable categories or
function tags. As in the example above, configuration information
is pulled from the BSim server and signatures are generated from the Ghidra Server
executables. If the <SPAN class="bold"><STRONG>config=</STRONG></SPAN>
executables. If the <SPAN class="bold"><STRONG>--config</STRONG></SPAN>
option is used, assuming the template it specifies is the same one used to create the
database and there are no executable categories or function tags, the BSim server
does not need to be running.</P>
@@ -199,7 +199,7 @@
<TABLE border="0" summary="Simple list" class="simplelist">
<TR>
<TD><CODE class="computeroutput">$(ROOT)/support/bsim commitsigs
postgresql://localhost/repo /xmldirectory [override=<EM>ghidraURL</EM>]</CODE></TD>
postgresql://localhost/repo /xmldirectory [--override&nbsp;<EM>&lt;ghidraURL&gt;</EM>]</CODE></TD>
</TR>
</TABLE>
</DIV>
@@ -215,7 +215,7 @@
"emphasis"><EM>repository</EM></SPAN> and <SPAN class="emphasis"><EM>path</EM></SPAN>
associated with it in the form of a <SPAN class="emphasis"><EM>ghidra://</EM></SPAN> URL
that was recorded when the XML files were generated. This path can be overridden with the
optional <SPAN class="bold"><STRONG>override=</STRONG></SPAN> parameter where a revised
optional <SPAN class="bold"><STRONG>--override</STRONG></SPAN> option where a revised
Ghidra URL may be specified.</P>
<P>The <SPAN class="command"><STRONG>bsim commitsigs</STRONG></SPAN> command can be
@@ -526,21 +526,21 @@ public void adjustTags(Address myaddress) throws Exception {
<TABLE border="0" summary="Simple list" class="simplelist">
<TR>
<TD><CODE class="computeroutput">$(ROOT)/support/bsim delete <SPAN class=
"emphasis"><EM>bsimURL</EM></SPAN> md5=<SPAN class=
"emphasis"><EM>&lt;bsimURL&gt;</EM></SPAN> --md5&nbsp;<SPAN class=
"emphasis"><EM>7abf...</EM></SPAN></CODE></TD>
</TR>
<TR>
<TD><CODE class="computeroutput">$(ROOT)/support/bsim delete <SPAN class=
"emphasis"><EM>bsimURL</EM></SPAN> name=<SPAN class=
"emphasis"><EM>&lt;bsimURL&gt;</EM></SPAN> --name&nbsp;<SPAN class=
"emphasis"><EM>...</EM></SPAN></CODE></TD>
</TR>
</TABLE>
</DIV>
<P>In the <SPAN class="emphasis"><EM>md5</EM></SPAN> form, you specify the 32 character
<P>In the <SPAN class="emphasis"><EM>--md5</EM></SPAN> form, you specify the 32 character
hex representation of the md5 hash of the executable, which should identify it
uniquely. Using the <SPAN class="emphasis"><EM>name</EM></SPAN> form, there is the
uniquely. Using the <SPAN class="emphasis"><EM>--name</EM></SPAN> form, there is the
possibility that the name is not unique, in which case the command will fail.</P>
<P>If a unique executable is identified, its metadata record will be removed, and the
@@ -580,11 +580,11 @@ public void adjustTags(Address myaddress) throws Exception {
<TABLE border="0" summary="Simple list" class="simplelist">
<TR>
<TD><CODE class="computeroutput">$(ROOT)/support/bsim generateupdates
&lt;ghidraURL&gt; &lt;/xmldirectory&gt; config=&lt;config_template&gt;
&lt;ghidraURL&gt; &lt;/xmldirectory&gt; --config&nbsp;&lt;config_template&gt;
[--overwrite]<BR>
$(ROOT)/support/bsim generateupdates &lt;ghidraURL&gt; &lt;/xmldirectory&gt;
bsim=&lt;bsimURL&gt; [--commit] [--overwrite]<BR>
$(ROOT)/support/bsim generateupdates &lt;ghidraURL&gt; bsim=&lt;bsimURL&gt;<BR>
--bsim&nbsp;&lt;bsimURL&gt; [--commit] [--overwrite]<BR>
$(ROOT)/support/bsim generateupdates &lt;ghidraURL&gt; --bsim&nbsp;&lt;bsimURL&gt;<BR>
<BR>
$(ROOT)/support/bsim commitupdates &lt;bsimURL&gt;
&lt;/xmldirectory&gt;</CODE></TD>
@@ -596,11 +596,13 @@ public void adjustTags(Address myaddress) throws Exception {
stripped down metadata XML files for every executable contained within the repository
folder specified by the <EM>ghidraURL</EM>. Just like the <SPAN class=
"bold"><STRONG>generatesigs</STRONG></SPAN> command, it can take an optional <SPAN
class="bold"><STRONG>config=<EM>config_template</EM></STRONG></SPAN> parameter, which
allows the command to execute without the BSim server running. It can also take an
class="bold"><STRONG>--config&nbsp;<EM>&lt;config_template&gt;</EM></STRONG></SPAN> parameter, which
allows the command to execute without the BSim server running, otherwise a <SPAN
class="bold"><STRONG>--bsim&nbsp;<EM>&lt;bsimURL&gt;</EM></STRONG></SPAN>
parameter is required. It can also take an
optional <SPAN class="bold"><STRONG>--overwrite</STRONG></SPAN> parameter, causing it
to overwrite any previously generated XML files. If a
<STRONG>bsim=<EM>bsimURL</EM></STRONG> is specified with the <STRONG>--commit</STRONG>
to overwrite any previously generated XML files. If the
<STRONG>--bsim</STRONG> option is specified with the <STRONG>--commit</STRONG>
option updates will be committed directly to the database. A BSim database commit is
always performed using the specified <EM>bsimURL</EM> if an <EM>xmldirectory</EM> is
not specified.</P>
@@ -635,7 +637,7 @@ public void adjustTags(Address myaddress) throws Exception {
<TABLE border="0" summary="Simple list" class="simplelist">
<TR>
<TD><CODE class="computeroutput">$(ROOT)/support/bsim dropindex <SPAN class=
"emphasis"><EM>bsimURL</EM></SPAN></CODE></TD>
"emphasis"><EM>&lt;bsimURL&gt;</EM></SPAN></CODE></TD>
</TR>
</TABLE>
</DIV>
@@ -646,7 +648,7 @@ public void adjustTags(Address myaddress) throws Exception {
<TABLE border="0" summary="Simple list" class="simplelist">
<TR>
<TD><CODE class="computeroutput">$(ROOT)/support/bsim rebuildindex <SPAN class=
"emphasis"><EM>bsimURL</EM></SPAN></CODE></TD>
"emphasis"><EM>&lt;bsimURL&gt;</EM></SPAN></CODE></TD>
</TR>
</TABLE>
</DIV>
File diff suppressed because it is too large Load Diff
@@ -245,7 +245,7 @@ public interface FunctionDatabase extends AutoCloseable {
public static boolean checkSettingsForInsert(DescriptionManager manage,
DatabaseInformation info) throws LSHException, DatabaseNonFatalException {
if (manage.numFunctions() == 0) {
throw new DatabaseNonFatalException("ls ~/junk" + "");
throw new DatabaseNonFatalException("Empty signature file");
}
int res = info.checkSignatureSettings(manage.getMajorVersion(), manage.getMinorVersion(),
manage.getSettings());
@@ -37,6 +37,7 @@ import ghidra.features.bsim.query.client.tables.ExeTable.ExeTableOrderColumn;
import ghidra.features.bsim.query.description.*;
import ghidra.features.bsim.query.protocol.*;
import ghidra.features.bsim.query.protocol.ResponseDelete.DeleteResult;
import ghidra.framework.Application;
import ghidra.framework.client.ClientUtil;
import ghidra.framework.protocol.ghidra.GhidraURL;
import ghidra.program.model.listing.*;
@@ -184,6 +185,10 @@ public class BulkSignatures implements AutoCloseable {
insertreq.path_override = GhidraURL.getProjectPathname(ghidraOverrideURL);
}
loadSignatureXml(file, insertreq.manage);
if (insertreq.manage.numFunctions() == 0) {
Msg.warn(this, file.getName() + ": does not define any functions");
continue;
}
if (insertreq.execute(querydb) == null) {
Error lastError = querydb.getLastError();
if ((lastError.category == ErrorCategory.Format) ||
@@ -646,7 +651,7 @@ public class BulkSignatures implements AutoCloseable {
establishQueryServerConnection(false);
ExeTableOrderColumn sortEnum;
if (sortCol != null) {
sortEnum = ExeTableOrderColumn.valueOf(sortCol);
sortEnum = ExeTableOrderColumn.valueOf(sortCol.toUpperCase());
}
else {
sortEnum = ExeTableOrderColumn.MD5;
@@ -961,11 +966,12 @@ public class BulkSignatures implements AutoCloseable {
protected File establishTemporaryDirectory(String xmldir) throws IOException {
File dir;
if (xmldir == null) {
String tempDirString = System.getProperty("java.io.tmpdir");
if (tempDirString == null) {
File tmpDir = Application.getUserTempDirectory();
if (tmpDir == null) {
throw new IOException("Could not find temporary directory");
}
dir = new File(tempDirString, "bulkinsert_xml");
dir = new File(tmpDir, "bulkinsert_xml");
deleteTemporaryDirectory(dir);
}
else {
dir = new File(xmldir);
@@ -983,6 +989,9 @@ public class BulkSignatures implements AutoCloseable {
}
private void deleteTemporaryDirectory(File tempDir) throws IOException {
if (!tempDir.exists()) {
return;
}
File[] listFiles = tempDir.listFiles();
if (listFiles == null) {
throw new IOException(
@@ -1106,6 +1115,7 @@ public class BulkSignatures implements AutoCloseable {
if (manager.numFunctions() == 0) {
Msg.warn(this, program.getDomainFile().getName() +
" contains no functions with signatures");
return;
}
FileWriter fwrite = new FileWriter(file);
manager.saveXml(fwrite);
+1 -1
View File
@@ -3,7 +3,7 @@
# Command-line script for interacting with a BSim database
# maximum heap memory (may be increased)
MAXMEM=1G
MAXMEM=2G
# launch mode (fg, bg, debug, debug-suspend)
LAUNCH_MODE=fg
@@ -3,9 +3,8 @@
@echo off
setlocal
:: Maximum heap memory may be changed if default is inadequate. This will generally be up to 1/4 of
:: the physical memory available to the OS. Uncomment MAXMEM setting if non-default value is needed.
::set MAXMEM=2G
:: maximum heap memory (may be increased)
set MAXMEM=2G
:: launch mode (fg, bg, debug, debug-suspend)
set LAUNCH_MODE=fg
@@ -23,13 +23,13 @@ Non-shared projects are locked when open, and the lock will prevent the signatur
<pre><code class="language-bash">cd &lt;ghidra_install_dir&gt;/support
mkdir ~/bsim_sigs
./bsim generatesigs ghidra:/&lt;ghidra_project_dir&gt;/postgres_object_files bsim=file:/&lt;database_dir&gt;/example ~/bsim_sigs
./bsim generatesigs ghidra:/&lt;ghidra_project_dir&gt;/postgres_object_files --bsim file:/&lt;database_dir&gt;/example ~/bsim_sigs
</code></pre>
<ul>
<li>The <code>ghidra:/</code> argument is the local project which holds the analyzed binaries.
Note that there is only one forward slash in the URL for a local project.</li>
<li>The <code>bsim=</code> argument is the URL of the BSim database.
<li>The <code>--bsim</code> argument is the URL of the BSim database.
This command does not add any signatures to the database, but it does query the database for its settings.</li>
</ul>
@@ -22,12 +22,12 @@ To generate the signature files, execute the following commands in a shell (adju
```bash
cd <ghidra_install_dir>/support
mkdir ~/bsim_sigs
./bsim generatesigs ghidra:/<ghidra_project_dir>/postgres_object_files bsim=file:/<database_dir>/example ~/bsim_sigs
./bsim generatesigs ghidra:/<ghidra_project_dir>/postgres_object_files --bsim file:/<database_dir>/example ~/bsim_sigs
```
- The ``ghidra:/`` argument is the local project which holds the analyzed binaries.
Note that there is only one forward slash in the URL for a local project.
- The ``bsim=`` argument is the URL of the BSim database.
- The ``--bsim`` argument is the URL of the BSim database.
This command does not add any signatures to the database, but it does query the database for its settings.
## Committing Signature Files