>> I'm technical director, Matt Hastings, one of our consultants, both of us has done
response investigation at Mandiant, for myself upwards of five years,for matt upwards of
three years. Apologize if we sound horse. I have been in Vegas since Friday of last week
teaching and presenting. This is last of my voice that's left. Hopefully it will carry through
the hour. Thank you for coming. We will talk about investigating power shells attacks. Not blue
shell. If that's what you are looking for in the wrong presentation. The reason we did
this research ‑‑ we continuously are seeing attackers adopt interestingly novel approaches
to compromise and persist an environment. We had seen an increasing power shell in some
targeted attacks we investigate starting about 18 months to two years ago, very sporadic,
nothing consistent, nothing I would call a trend at the time. Earlier this year matt and I
worked the case, targeted compromise of the fortunate 100 cases, spent four months
investigating in this environment. This victim VPN space have been compromised for
over three years. Attacking main administrator during that time. They had free access to the
majority the environment, but the way they have conducted the majority of their of command and
control when we were watching was from their own originating attacker system through the VPN,
targeting the victims work station and server using a combination of stuff we see
often, like schedule tasks and stuff not seen quite so often, power shell remoting and copying
and executing power shell scripts on the target systems. This turned out to be
forensically challenging. You don't get to see what's on attacker's system. When you
conduct attacks like power shell remoting over the network, it's difficult to see what was done
on the host on the end point side if it's in memory. This lead down the path of doing
research and that's why we're here today. Of. We're not going to go into a power shell 101,
way beyond the scope of this talk. Suffice it to say power shell is powerful. A command
line and scripting environment in Windows takes the functionality see in batch
scripts or other scripts Windows supported and rachets it up another level. You can interact
with Windows API and registry and you can do cool stuff access the entire dot net frame
workload and reflectively load and execute DLL and memories. This makes it useful for system
administrators and developers for using it legitimate purposes. It also means
attackers can use it and do neat stuff witness. Developers have built penetration testing and
attack frame that use power shell. PowerSploit is probably the best known written by a
number of guys, Bell, Joseph, Mack who works with us at Mandy FRI. PowerSploit does
recognizance, remote code execution,its got a module called invoke Mimikatz that will
take the mimikatz code and in memory run it on a remote system without ever touching disc and
do what mimikatz does and other modules. Other tools like MetaSploit which has power shell
based modules for over a year now, veil power view, more and more attack framework are
integrating power shells are integrating power shell into their tool kit. Targeted
attackers learning how to use it, attack and testing tools using it. Mass malware use power
shell. The first power shell mass malware was shortly after the creation in 2006 called
power worm. A proof of concept didn't do much but now mass malware use power shell. Trend
lab has reported a couple examples in the last 12 month. A classic ransom where sample that
encrypted all your file and demanded payment. The bottom line attackers of all
disciplines are learning how to use this stuff. Instead of coming up with an investigation
methodology specific to one attack frame work or cool or malware, we took the lowest
common denominator. What is the fundamental way to interact with a system and forensic impact
does that have? The three scenario exercise executing a power shell local host, using
power shell remoting feature which using a protocol WinRM, Windows remote management
interact with a remote host and persistent power shell, taking power shell code and configuring
system to automatically execute upon log in or boot time. Each scenario you run test case what
impact on registry on file system event logs, memory and monitoring the memory, network
traffic. A few assumptions, the fist that we make is that the attacker has local or domain
administrator access to the target system. You may be thinking, isn't that cheating,
what we're dealing with, when we look at this is post compromise scenarios. I don't care about
the initial exploit, I concern has a foothold in the environment and achieving
mission, moving host to host, doing recognizance, stealing data and whatever else brought
them to that environment in the first place. Almost every Windows environment the attacker
gains administrative permission quickly. You can find the other stages without getting
compromised and kicked out. That's the context we will be looking at all these
environments. A quick version reference, there's a lot of different versions of power
shell. I think 4.0 is current, 5.0 is in the work. in most Enterprise environment, you will
see 2.0 because thats whats been installed by default in Windows 2007 and viSta late and Server
2008R2. Windows 8 power shell 3 by default, Windows 8.1 includes 4, but we dont see corporate
environments adopting either of those platforms for quite some time. The down side is cool
things in power shell 3 that myself and Matt will talk about later, won't be available by
default and therefore won't be installed in most environments. You have to apply an optional up
date in the Windows management frame work to gain access to later versions of power shell
and Windows version of Windows. Let's begin by talking about memory analysis. In some ways
this is the worst case scenario. An attacker interact with targeted host, use power shell
remoting. Worse case scenario because it's volatile, the session end, come back to system
in minute, hours days, and based on a lot of factors we want to understand what would be left
behind assuming the system had not been shut down or rebooted and how long could you find and
recover that and how much could you get out of memory. To understand this, we need to
understand the process hierarchy that power remote using. What happens myself as client
initiates power shell remoting on target systems memory? The way it work lets say i use the
power shell command like the invoke command to run evil .exe and presumably evil.exe is
already on target system. svchost.exe service hosting process that is running the
decom launch service an instantiates an instance of wsmprovhost.exe and then that
instance of wsmprovhost in turns, executes evil.exe so evil.exe parent process will be
wsmprovhost and svchost will be that wsmprovhost's parent process host. In contrast, If I
use power shell remoting to execute power shell native command, in other words, just
command or code that is solely resonant within power shell language based, what happens is
wsmprovhost spins up again. You don't see a child process power shell dot exe spawned. The power
shell code runs in context of wsmprovhost. You don't see a child process when you do
remoting like this. So what's in memory? If you go to memory and you dump memory right when a
power shell session, remoting session occurring on victim host, you can reconstruct almost
complete command from the wsmprovhost memory space. The problem with that is it
terminates at the end of the session. Meaning if I do one power shell remoting command,
the moment the command complete, that wsmprovhost terminates, all the cool evidence is totally
gone. From a practical standpoint as an analyst or investigator you will almost
never get to the system in time to recover that process memory. That kind of brought us back to
the drawing board to see where else in memory does this stuff hang out? What we found is that
another instance of the service hosting process running the Windows remote managing service,
broker power shell remoting, contains remnants of the soap objects that power shell input
and output martial-led into when you do remoting. This may persist after the end of a
remoting session maybe up to reboot. We found bits and pieces of the same data persist in the
kernel and the page file. We suspect that that is probably artifacts of page, tools and
memory and therefore not something you can reliably expect to always find. We will
not read this in detail. We put together a table here of source of evidence in memory and how
long you can find stuff after the fact on the system. Your best bet is the winRM instance
of Svchost. If you dump memory, assuming that the attacker is the only one thats interacting
the system through remoting, and assuming that the system hasn't rebooted, you can recover input
and output from power shell sessions. It look like this. I apologize if the colors are hard
to read. Here's an example of us running through remoting of power shell command that does
echo test string PS session through output file. Amidst this giant blob of ugly xml, you can
see an RSB command tag and the full command we submitted in the command line. We were able to
recover that from memory many, many hours after that already had executed. A more interesting
example used the power sploit invoke mimikatz madule again for remoting. We used the technique
where you download through power shell the script over the Internet, and then dynamically
execute it. So nothing touches this memory on the victim system. Again, we can see in the
inset a complete command line of everything executed to pull and run the invoke mimikatz Power
sploit module. It's possible to use this. Yes, this is painful. This is kind of like worst‑case
scenario. You can recover evidence. Will you have to work for it. The biggest problem is
since we're looking at fragments of the soap and the power shell remoting protocol, xml objects
used when do remoting, not like well structured objects retained in memory because being actively
used. It's portion of memory no longer actively allocated. The bottom line you can do string
searches. You can find RSP command,RSP command line the strings and hone in on the
commands. You will have to sift through a lot of garbage. In summary, timing is kind of
anything, everything rather, that's always the case with memory analysis but especially
with this because there is no proper place at which in memory all of power shell history is
retained forever. It has to be constructed from what's left behind in the Windows R M memory
space. Taken then things like system up time and memory utilization can impact how
likely you are to recover these types of art facts. I will turn to Matt? >> You look at similar
scenario Ryan talked about attacker interacting with system, and we also looked at
logs in power shell locally. Each scenario try to answer three questions. Which event log
specifically are activity? This will be event log that are directly writing power shell
activity. What you will see in the next slide is event logs that power shell writes to
directly. It doesn't necessarily cover the event logs that may contain data. For example in the
remoting example, an attacker logs in, you will expect to see a log in. That wasn't covered in
research. As looking for each resource, logging detail, ideally get full command history
of everything executed on the system. What will we see? Specifically some of the
distance between power shell 2 and 3. So we identify five specific logs that may contain
power shell commands or may contain some relevant information related to power
shell execution. The next two slides will walk through what happens local versus remote
scenario. Starting with the power shell event log itself, we identified two event ids for
104, three contained some bit of relevant information. Not ideal if you are looking to fully find
out what was executed. but in the EID 400 seeing the engine state changes from none to say
available, indicating that power shell started. Looking at generated time of event, power
shell was run on the system at this time. then in EID 403, done running. You can two times and
command between X and Y. The host name consult host indicates a local session versus a remote.
We will show you what a remote session looks like in a following slide. A power shell
operational event log and power shell 2.0 doesn't contain any information. We did find one
that contained the same information that power shell starting up. You can get that
from the same event he spoke about previously. Power shell 3.0 and greater, it write for
error messages. A script failed to run, we get the full path of the script. In a remoting
session session, if looking at remote host, you can get the host name and I T address of the
system attempt together be accessed. For power shell analytic logging, those not
familiar, analytic logging is disabled by default. Debugging purposes and not for long‑term
retention. We did enable it and find that power shell contain relevant information. You can
see here in the 79, 37, the test dot PS 1 scrypt was run. We can see a power shell script run.
You can run a native within the same event I D and finally, a dropper I D. Run that from the
power shell console, that will be recorded here as well. What you don't get from here is any
command or argument, you kind of get like half way there and then you want to know how were these
configured and there's better ways to record this information. An analytical logging in this
case is probably not worth it. Moving on to remoting section here back to the power shell
log, EID6 the accessing system, we can see that the WS man session is starting. Looking at
this, you can say from the system of remote session starting connecting to another
host. And those not familiar, W Sman manager protocol used by win R M which is what power
shell uses for remoting. On the access system, we have E I D 400 and 403, in this case we see the
serve remote host as a host name, which indicates that this is a remote session versus a
local session. In the remoting situation you are using the winRN service, specifically here
in EID 169 you can identify who the user that logged into the system was, if you don't have
logs going back that far. The security protocol that was use Ford authentication. Similar to
the power shell log, the time frame of remoting activity, the client request for the shell and
the delete shell. That appears in multiple event logs. Now going back to the power shell
analytic log. As far as 2.0 is concerned, which Brian mentioned, it is probably
prevalent. I got to pause. Yesterday a mispronounced prevalent Back on track. In the
analytic, way of capturing command execution and responses. Now to problem is, and well will
get to this. The actual content written in hex encoded format. Looking at the log itself
doesn't give any information. What you will find is that every piece of information, and
everything that is sent back and forth between the two host will record it. The initial protocol
is recorded in the log hex producing format. You have to dig through it if you want to
find what was run. Here is an example of the event log. Here an object was received on
company man, again, it's in a hexing encoding format. You can't tell what happened. If we
decode it, you can see here that the lines here. You can see it. The command get char item in the
script, no, not, you can see the argument pass. You can see ‑‑ Looking at the analytic log now
with a response that is sent back, a single response that's sent back to the request in the
system. In power shell we run a built‑in command, everything that comes back is in the form
of an object. Each object returned from this command is written as a single event in the
event log. If you run this get trial, it will create enough in the event log to mass the number
of items within the C directory. It generates a ton of events and you have to parse through
individually decode and map back to what was run and what was responded. In 2.0, it's not
practical to do this. There are some other options. The first one is some of the profiles that
power shell has. Power shell has multiple profile options where you can configure code to be run
at any time that power shell locally starts up. Here an example of global profile. You
can start record transcript command input and output. You can write out to an event log
format if you prefer. We have seen some companies put together custom codes overrides the
default prompt so anything typed in spit out to a file. Some of the limitations here are like I
mention, this is on for power shell execution. It doesn't do anything over remoting. Also,
you can launch power shell by saying no profile flag, so none of this codes are run. Again,
it's a solution not the best answer, but it is something that can help you out. Also, if you
are using app blocker, it has an option for enforcing scripting rules. You can see here, okay, a
script was attempted to run at this time. Here's the path to it. Even if not enforcing, it's
another option if you are running app blocker. Moving out of power shell 3.0, the best
option we found is module logging. What it allows is to get that clear text specific
command input and output in clear text for anything being run through the shell. Both
locally and remotely. We will show you in further slides. Here we ran a more complex command in
the past. We said get trial item of C attempt directory recursively only filter back
files with the .text extension and do a basically a select screen in power shell for the
same pass word. The top item here, you see we captured the command sent to the system and
have all the arguments being parsed out and available in a clear format and at the bottom
you see the response or see the file that it hit on and the line where it found the file, found
the hit. Now, this can be slightly problematic when you run a script. Each line and
script, each command in the script will generate own power shell script. We ran the invoke
mimikatz module in power shell and doing over 1200 events in the event log. A large number.
We can't parse through. When we got to the actual response, we found it written in clear text
mimikatz output in the event log. so you can see all the password hashes here clearly
written in the event log. >> For this last bit were gonna focus on, we wanted to talk about the
ways in which power shell code configured to automatically run in the system. Attackers like
persistence, if you are look doing something like back door or key stroke or car data track,
or track recording, anything continuously execute on a system and survive reboot and shutdown,
you need to find persistence mechanism. This is understood. There's numerous techniques that
Windows provide well‑documented from registry, auto start point, window services, thing like
recurring schedule tasks, use of the start‑up folder, if you think about it, persisting power
source, making power dot exe run with argument of the script. All of these techniques very well
understood. We will not spend a lot of time talking about them because folks have done plenty
of research on that in the past. What we will spend time on is per these tense via window
management. One of the reasons nor that is the power sploit tool kit includes this as one of
the mechanism that you can use to persist the code of your choice. We thought that made it
a target. We seen a packers use W M I persist, not necessarily power shell but other things
like C script or J script, the other built in scripting component in Windows that
similarly running native and code. W M I is friendly for that scenario. To understand that, we
need to step back and explain how W M I works. This is a big mess. I will give you a high
level. It's this inventing infrastructure. It has this notion of event filter,
consumers and binding between the two. An event filter is a query continuously run against
the state of the system. When it's satisfied, it fires the consumers that bounds ‑‑ and the
consumer runs something. All of these things can be set with power shell using set wmi
instance command. And also as we'll show you later enumerated power shell. Event filter, as I
said is the thing that causes the consumer to trigger. Written in Wql. Has a simple example. We
have an example of wql that will fire when the system up time between 240 and 325 seconds;
basically, a roundabout way of saying, fire upon startup. The next one is an example where
instead of relative time, it says at 12:00 exactly fire. So that will satisfy the filter.
Two different ways to basically achieve the same thing, continuously run something on an
agreed upon interval. The consumer is what runs when the filter is satisfied. In this
example we're going to execute power shell. The trick is execute power shell to load
what? Where does the evil code come from? You have a few options here. The thing that
power sploit does by default is it puts the code in system wide or user profile. As Matt
mentioned earlier, load automatically when power shell executes. If I create a system
runs power shell without any options, it loads the profile. By virtue, it loads the
malicious code. Simple technique and works nicely. The other thing is add your malicious
power shell code to argument supplying to power shell exe and build into the consumer's
command line. In that case only limited by the command line length limit Windows enforces
something like 8,000 characters. That isn't too bad of a constraint. The example shows
you can run compressed power shell code and decompress and execute on the fly. You can fit
8,000 characters. The easy solution to find malicious wmi filter and consumers that run
this is through the use of power shell. The get wmi company man lets you filter consumer and
binding between the two. We do investigation in a lot of large environments. We pulled 50,000,
100,000 environments to get an idea of what is and what is not. What we have seen is that it is
not common to have event consumers that execute power shell. That doesn't mean in your
environment it may not be legitimately used for administrative purposes. The
bottom line is base line environment easily and determine what consumers and filters and
legitimate and look for anomalies. The other thing you can do is look at the file
system at the host which this occurred. This is difficult because wmi is stored in the
wbem repository. This directory C Windows system 32 wbem repository. The files that keep
all of the wmi objects in event filters and consumers, mainly objects.data is the key file. It
is this undocumented proprietary format no documents whatsoever. Some engineers looked at
briefly. A big ball of hurt. They don't want to get into. The bottom line is if you had a dead
disc image from a host, there's no good way to host it purchase. You can do strings against it
which can kind of good. Strings will give you the event filter and consumer names. If your
attacker was foolish enough to create my bad ass event filter, you would be able to find in
object data. You get the command line. If you discover on a host this technique is in use, the
attacker is referencing power shell with arguments, you can see that in strings in
objects.data. If you are looking at things I'm examining a system, did these files get
modified in the time frame attack activity? It will not work. Windows is up dated
constantly. They change all the time. Finally, you should definitely review all the
profile .ps1 file on a system that you suspect has been hit. In most environment, you will
have an empty file. So quick and easy to check the profile. You might as well look at it.
Anything in the profile will automatically execute unless you supply the no profile argument
to the power shell command line. Why not check it for malicious code? Finally, the registry,
this was kind of disappointing, like many thing in Windows creating a wmi filter, a
consumer lead to some sort of change for registry forensically examined. That was in this case.
The only exception was depending on the filter you used to create the condition to fire consumer,
registry team might be created to a provider that satisfied that query. The long and short
is the example power sploit uses run this on this exact hour and minute, it was like 12:00, in
the example. Creating that filter registers this key for win 32 clock provider, by
default this does not exist in Windows environment. You have to create a wmi filter that uses
that as its provider. From all the enterprise scale investigations done, I have not
seen this key except when an attacker is using this technique. Of course there's a
hundred million ways you can create wmi filters that don't use that specific provider. This
isn't meant to be comprehensive. It was an interesting part of the fact of default behavior of
PowerSploit itself. Other sources of evidence, if you use the system internal tools, which
are fantastic, auto runs was updated a month or two ago. Version 1d will enumerate and
displays all wmi filters and consumers in an easy to read format. I suggest checking that
out. In memory, we didn't spend a terrible amount of time examining this. That was mainly
because the memory space of the wmi relevant processes are noisy. We didn't find any
reliable way to recover evidence after the fact. Event logs do have some wmi, so you can enable
wmi trace logging. If an attacking register a malicious wmi, it would generate an event.
Windows does this stuff behind the scene. The one event will be buried amongst thousand events
during system up times and roll off the log quickly. >> Just to wrap up. We didn't cover every
source available. We have released a white paper now public, in a draft format. The
final version out next week. It has more detail and some of our research and findings. A few
other places if you look, if all the other ones we have gone through turned up nothing,
looking at Windows work station and a power shell pretext file created, that can contain
information on either a power shell scriptures line or power shell certain files of interest
to you. Now only caveat only for local execution of power shell. No remoting here. Also, there's
a single registry that I wouldn't say relevant, the execution policy setting. By
default power shell doesn't let you execute unassigned scripts this is just to prevent people
from double clicking bad scripts. Everything else in power shell, you can bypass
command line with the execution bypass flag. One situation, the attacker modified registry key
to allow unsigned scripts rather than bypassing on the command line. I don't know if tired of
doing it, they did modify this key. Network traffic analysis, since power shell using the win
rm service, looking at the winrm port so port 5985 and 5986 use http and https respectively. The
pay load for power shell is encrypted regardless if its http. If you are using http, you
can pick up header information power shell version. In that case, mentioned previously, the
attacker is using a different version of power shell than what they use natively in the
environment. They actually wrote some ‑‑ did some network monitoring power shell based on
the version in the headers. And then looking for anomalies. Really, lessons learned and
recommend upgrade module logging possible. Move to a power shell a version 3 or later and
implementing module is the best solution. Beyond that and looking at high level, base line
legitimate powershell usage usage can be key in identifying anonymous and malicious
powershell usage. What is execution policy setting and does it change? If so, why? How
do you name the power shell experiment and where do you put them and looking for other
directories that may not match? Do you use remoting? Is remoting enabled by default or only on
certain system and base lining that usage? What users use it and from? Is it only admin or
serve admins and work from similar sub net and base lining and saying this is what good
power shell looks like in my environment and looking for anomalies through the method we
described earlier. >> The bottom line exception of module loggng, no perfect solution. There's a
lot of attackers continue to do it. Taking that step in advance before an incident understand
how being used can save a lot of leg work in the future when you may have to respond. The thing
we find is during your investigative process, power shell isn't the lead. It's some
other artifact activity on the system. As doing forensic and time line analysis and see these
thing we presented today, you may get indication that attacker was using power shell. If you
didn't node to look in these places and memory, you might completely miss that's part of
their tool kit and therefore they might be able to successfully use it without
being detective in post compromise. We want to acknowledge Matt, co author of
powersploit, done a lot of cool work on attack research with power shell. As has Joseph,
Chris Campbell, Microsoft has reached out. I want to thank Lee Holmes. The folks at Microsoft
are aware of folks using it. It's under the context of attacker who is already in the
administrator on system. All bets are off from security standpoint. They are working to
make it more audit friendly and easy to monitor. There will be some cool stuff coming down the
pike make it easier for those running in their environment. I think good thing will come out.
Thank you for your time and attendance here. If you have any questions, it looks like we have
a couple of minutes. Step up to the microphone. We will answer them. You want to talk to us
directly, catch us after the talk or email us or hit us up on Twitter. At this point I will
stop for any questions. Sure. Could you go up to the mic please so it gets caught in the
recording. We will repeat it. >> AUDIENCE MEMBER: [Indistinct speech] >> The question
recommended powershell 3, not up date to 4? Not that I know . You can just as well update to 4.
They don't make it automatic. There were a couple of things that were deprecated between 2
and 3 ,they're exceptionally minor, for the small pool of people using power shell for
exchange event management, you may not want to break things. I would check whether the thing
you are using in 2.0 not broken. 2 to 3 was the major change. Does that answer your question?
ANNOUNCER: Yes. >> Any other questions? Once again, if you want to talk to us directly, we
will be here. You can reach out to us via email. You guys were great. Appreciate you having us
here. (Applause) [End of session] "This text is being provided in a rough draft
format. Communication Access Realtime Translation (CART) is provided in order to facilitate
communication accessibility and may not be a totally verbatim record of the class." PAGE PAGE
20