So I will get them started but I have to go on record just being so honored that the stage
is staring Dustin Hoffman presenting for us today. Give Dustin a round of applause. >>
Thanks that never gets old. I'll tell you. >> [Off mic]See, if you were not a speaker, I’d say
fuck you… >> You can still tell me, dude. You can tell your friends later. The only people
that seem to get that are really old bank tellers or people not from this country. Like oh,
that's a famous actor. Hey, thanks for coming everybody. I appreciate you being here. You
can be at any other talk right now but you chose mine so I am thankful for that. This is What
the Watchers see, Weaknesses in Municipal Mesh Deployments. I’m Dustin Hoffman. >> And I'm TK.
>> Wait. I have to talk about me. I am my own and also a senior engineer at an IT
services firm and principal of various other concerns and I also like to cross fit so we can
compare times and this is my colleague TK. >> Yeah, I guess I'm a senior engineer. We don't
do the title thing too much but that one seems ‑‑ we make them up pretty much every time. I
don't have a whole lot to say about that apart from being the principal of research. >> How
this got started for us was ‑‑ excuse me for a second. Oh shit. >>Excuse me for a second we have
a tradition here at DEF CON. These are first time speakers. >> We have one first time
speaker so they're both taking it. But the tradition is we have a ‑‑ we welcome new speakers by
allowing them to enjoy a shot with us. >> I'm going to go on record. You do a much better job
of pouring than Proctor. >> Cheers gentlemen. >> Cheers. [Applause]. >> Oh, thank you. >>
Thank you guys. >> Carry on. >> Thanks. So how this all got started for us obviously in the
post 911 world a lot of municipalities are taking Homeland Security money or any
moneys available when they have shrinking budgets and using technology and provide security
or oversight or surveillance throughout their city. We started to see these camera pods
and there will be pictures later. Being the curious souls that we are, we’re curious about
that. In certain places, my phone, it would be like hey, there’s an open WiFi network
available. And they had interesting names like Police Department [laughter] And this
is before the era of you know, FBI Surveillance Van 004. This is 2008, 2009. >>So while
drunkenly hanging around downtown one night a few friends and I were wondering around and
we find this fountain. I decide to climb on top of it which may not have been the wisest thing
in the world but I did that and here I am acting like a clown and a voice came from above and
said, this is the police. Please get off the fountain. And it sounded like it was about the
150th time the guy said that. I see, oh, of course the camera pod Dustin was talking about.
And I don't know, I guess I wasn't really all right with being observed so it put me on
wanting to look at these things and see what it's all about. You can see the network available
even just walking by and a number of things we'll cover. A little creepy. So one of the
first things I thought of after thinking about the fact that even in a relatively small town
around 70,000 people by no means a large city, I thought of this article that I encourage you to
read on Wired, called Transparent Society. Has anyone read it,Transparent Society?
This short story described, two societies. Both were surveyed. In one basically the authorities
have the camera. In the other every citizen has equal access to the video and it's an
interesting experiment. One of the things I think civil libertarians and people like
myself are concerned about with this kind of thing is the information is asymmetric and
certain privileged people have access to it but people like us don't. There's no audit trail.
We don't know how people are using it. So those are some issues that we'll cover along
with the vulnerabilities. You might ask why are municipalities deploying such networks. Once
you deploy a network it's like building a road. All kinds of other fantastic things can
spring up alongside it. It's like a utility. Cities are using them to obviously monitor
traffic in intersections and on main and other roads. Monitor public spaces. They have two way
audio. I know in the city, the specific city that we'll be talking about in our talk today
they used infrared sensors to monitor parks after hours. So if you go to a city park after dusk
you will get the same voice, city parks are closed at dusk. >> They have a real nice video
of some kid running away. You see him looking then scatter. >> Presumably that's what they're
looking for. Frequently in the RFP's for these things you will see cities talk about we have a
shrinking police budget, but it let's us be everywhere. Whether or not everyone is okay with
them being everywhere is another question. I think of the audience mated speeding ticket
robots if you will that they deployed in Arizona that the voters, you know, flipped out
about. Maybe rightly so and complained about. If you were speeding these devices were on
the side of the road. Your current speed is this an if you're going too fast it blinks
at you. This would read your license plate number if you were going too fast and mail you a
ticket. So that's one of the reasons the cities are deploying things like this. Implementation
is almost universally a mesh network. Why not a wire network? Nobody wants to trench. Like I
said the budgets are shrinking. To wireless technology a city council or other committee has a
dramatically lower cost. I don't know if you've been involved in stringing fiber. The licensing
process and the costs are substantial. It will easily exceed the cost of cameras and
what not. You can deploy much faster. Most of them operate on 2.4 gigahertz, 5 gHz. These are
not special. Who actually does the install? I think this will be a key part of our talk. You
have specialized companies that really focus just on municipalities and other kinds
of government organizations. One in particular Leverage IS they have special qualifications and
basically, basically they're able to fill out a long RFP. So these tend to become the prime
vendors in installs like this. They're not technical qualifications as you will see.
>> Not at all. >> First picture on their website is a smartly dressed police woman. Presumably
providing real‑time response to real‑time crime. Including climbing on fountains. If you
look at their Twitter site which hasn't tweeted in a year or two they only follow other police
agencies. Implementation hardware itself is not terribly interesting. Things you already
encountered. Maybe it's just packaged differently. Specifically we see a lot of
fire tight vendor. Bosch cameras, normal DVR's. Here is some of the equipment we picked
up off E bay. 2.4 giga hertz antennas here. They usually stick these in outdoor
enclosures. There's plenty available on E‑bay. We don't have a big budget for DEF CON.
You can even on E‑bay you can buy some units I don't know if you can use the license but they
have radios to operate on restricted public services networks. I think it's 4.9. So
information specific to Firetide. You should talk. >> Well basically they're just
little boxes and as far as the people using them are concerned in one end you plug in the
Ethernet cable and the other end and magic happens and the package makes it to the other
side. We did a small amount and I have to thank you guys from another company I worked out for
helping me out. Jordan Hays did indeed attend the conference. All these guys I don't know if I
could have done it without them. >> My favorite story is oh you want help for this? Oh for DEF
CON, sure. Like this one in a couple weeks. So thanks for that. >> Yeah they really pulled
it out. Well really honestly, you know, I'm losing my track here. We were able to find out a
little bit of information. It kind of looks like these things or using something like they're
using ‑‑ what am I looking for? Oh man. Automated source routing or reverse sort‑‑ I don't
remember the name of it. In any case where in every packet there's a header that contains
every single hop that the packet must take in order to go forward and they added on bits as well.
Signal strength to help with their routing so they can make sure it's a good system that
automatically heals itself. In this case I think they sort of relied on the fact that Automesh
is sort of a trade secret and I don't know if it's really trade secret but they deliberately
made it hard to understand. Rather than actually using security. Um, I don't know if
there's anything we didn't already say. >> If you use any embedded box, any small low
power system it's going to be similar to this. Really the only unique thing about Firetide as a
vendor is they have a patented proprietary protocol they call Automesh. It's derived from
other actual documented protocols. A large amount of our time was spent as far as we know
with the first decoding of the protocol. A full decoding would permit full inter-operability.
Typically you will see with these deployments 2.4 gigahertz, open standards in this case with
Firetide equipment and uses Automesh which is not documented. In all the city's
documents which you know get on the website 2.4 is used but it's not mentioned. They only mention
900 megahertz and 5 gigahertz. So it makes me wonder if you see the sloppy implementation they
did if they're aware the 2.4 gigahertz radios are turned on. When we talk about mission
general, if you don't have a clear understanding on this with them ‑‑ with any other network
let's say especially a WiFi or wired network you assume a client node let's say streaming
video my phone has a path to the switch or some other more centralized distribution point.
With mesh you don't. All the nodes are peers, this peer may be able to talk to the backhaul,
whereas this one can’t. You don't have to string fiber to the city. So here in the first
example with the wire you see all 5 outer nodes presumably are reliable, semireliable and
linked to the center node in a mesh nodes can route the traffic through other peer nodes. This
is actually if you are ever going to do anything malicious scroll the city's website
because the information they provide is incredibly useful if you had ill intent. This image
is right off their website. It documents the specific antenna replacement and buildings they
route off of. Is it terribly small? Oh it is. I'm sorry. Things you can't know before you
get here. My apologies for the small images. You see the camera pods described and all the hop
locations. From this you can derive which antennas go where. Like I said Automesh, not
documented. They claim they have 19 patents to date. There's things you can assume that it
will do; a lot of the things that all the other mesh protocols will do. Link from
node to node. From reading many, many technical documents we discovered it embeds things like
each node for a particular altitude, GPS coordinates preset during install, things like this
which can contribute to the idea that with Automesh you don't have to do as much pre
configuration. >> So we set up and when we're looking at the traffic that's actually going
out over the air it starts out with kind of standard ‑‑ I wish I could point at it but I can't.
You can see it beginning there. There's a standard issue, headers and all that stuff and
additionally there's an IP header which is standard and that points to ‑‑ oh I can
point. How about that? That points to the mesh IP which looks a little weird. As far as
I know he everything is normal about that. The protocol is set to 134 which is allegedly for
RSVP. I know what follows is not in fact RSVP. Soon then we move on to the header right here.
This is what actually follow that will last IP header and the Automesh header. And this is
what I think is the mesh ID. There are only very few. >> Automesh uses this to
specifically identify which mesh, say you have multiple meshes deployed in the same
geographic region, which mesh this node is a part of. This is part of the Automesh secret
sauce. This is one of the things you define prior to deployment. We'll talk about it later in
more detail we believe this is part of what most of these deployments believe provide some
of the security they think they have. Specifically that it's not documented. You can't load up
for example Wireshark and get very far because it chokes on Automesh. One of our goals was
to help that along. How many people use Wireshark? It's super helpful. Right? It's great. It
chokes on some of this. Specifically because there's no public definition. >> We haven't
entirely decoded it here but you can see there's a couple fields that are probably flagged.
Actually the next one is ‑‑ this right here is a unicast frame and there's just not very much
information. This is a broadcast frame and I think what they've done here is they've attached a
little bit of information but maybe some flooding for getting routes and stuff like that. And
you can see this right after the mesh ID the last bit is 1. I was not able to find any packets
that had that set at 1 that was not broadcast and you have units here. There are a couple values
here. This is probably some routing info. Then there's this chunk that comes here. This
looks like it might be IP addresses and this here are definitely the IP addresses of
external ‑‑ they match the external IP of that first IP header we looked at. So I assume
it represents the idea of the node that we're entering. Then it goes to a multicast address.
Conveniently they've left for us ‑‑ oops, conveniently they left a marker at the end that
you can use to sort of strip this out entirely which when they have it unencrypted which
they sometimes do. There's an easy way to skip all the auto you mesh stuff so you can look
like we were able to send a package across the stuff and pull it out of the air which
takes away their safety there then this is just an example of something. This is a CDP packet.
Not much to talk about here other than you can see that it’s in the clear. >> Our specific
case ‑‑ thanks for sticking through the background. I hate to skip ahead and not have you
appreciate what you see in these things. In this specific city you see we mapped out most ‑‑ a
lot of ‑‑ there's apparently ‑‑ how many cameras are deployed? >> 122 I think. >> And that's
not including what they have to put overall the school sites. The city and school district
apparently joined hands and they kind of aggregated all their cameras together. The primarily
collection here is downtown. This is kind of the downtown. Oddly enough from the time the
initial RFP was released in 2006 until 72 hours ago the entire system was unencrypted. So no
WEP, nothing. So friends there's a lot of things we can't know in life. Like one of the things I
try to figure out is is it actually supposed to look like a Wu Tang clan? The world may
never know. In this case I'm going to try to put myself as best I can from servicing other
IT clients and put myself in the shoes of the City Manager and maybe the people that did the
original install. My best guess is the Automesh alone would provide the security because it
was obscured. Because you couldn't just load it up and read it right off without
additional efforts. It was secure enough. I think this is part of the problem with when we
use words like security. I don't know about you but I think security is over qualified. I
think it was said best in the books. There's two kinds of encryption in it is world. The
kind that keeps your kid from reading your diary and one that keeps companies and
multinationals from reading your diary. At the end of the install the QA manager, I'm being
generous that there is one, they didn't look at the form and say is it secure? Yeah it's secure.
We can check that box. Or it's a matter of well I turned the knob and the door didn't open. If you
lean on it a little it will pop right out. It's kind of one of those things and I think as
taxpayers, especially with the next slides you want to cringe because this was all tax money.
This is something you and I came out of our pockets and here we are frankly ill served by ‑‑ you
know I was thinking of this the morning. I thought of the Obama care website. Shoddy government
contractors took a lot of money and provided not a great product. I feel like you will
have shades of that repeatedly as we go through the slides. But 72 hours ago WEP was turned on.
So yea for government efficiency. Right? Was it because of our talk that was ‑‑
I don't know. I'm going to say it was though. I'm going to take credit for it. Moving the bar of
security forward. One peg at a time. >> You should read these later. So here's what you'll
commonly see. These are the camera pods watching you. Here's a loud speaker. Right? Here's
the little all weather enclosures. When you see things like this I get nervous. I have
to wonder if, you know, those directional antennas will provide security. Right? They
only send the signal to one other place. I wonder if ‑‑ think about when you didn't know
as much as you know. If you did things looking back you would be like oh yeah that was a bad
idea. That didn't work out as well as I thought. I have moments like that. When I
started the company I run now I thought that was a really bad idea. Not at all safe or secure.
Maybe you got by by the skin of your teeth. That's what happened here. We have Automesh. Then we
can read the video screens. We have directional antennas. No one else is going to see the
Redland's police pop up. And there's one of the Firetides right there. Having vendor names
here, maybe not the best idea. They were installing another pod right by our office and they had
some vendor out. Like a bucket truck. And I stopped to take pictures because he had the door
open to this thing. And that guy got upset. I was trying to take pictures with my camera. Here's
the highest point in the city. This 8 story building. It has cameras ‑‑ this is the high
point they route it back to. You can see these things are popping up everywhere. This picture
makes the city look really bad. This is more cameras. More PA devices. I'm not sure what I
find more obnoxious is the cameras or PA things. This makes it okay though because of a
notice it's under surveillance by the police department. I think they have run afoul. >>
This was not sufficient. You have to have notifications a lot more places than they have them.
>> It's not exactly downtown either. It's kind of like on one of the ways that you can get
into downtown. But no where else. >> I've never seen this sign which is weird. So as it
was until 27 hours ago which is not fair that I got here and they changed it right before we
came. So I'm so smarting on that. Here are the security problems and potential issues
with having it as is. And when you have one vendor doing the install, I'm confident you will
look around, when they get a government contract they're going to tell everybody. It
furthers their prestige the small community they're in. Santa Monica has a deployment by
them. Other cities larger than us. So assuming like smart people like us do that
directional antennas and network protocol is not enough. At the minimum ‑‑ and I'm going to date
myself but I was saying this back in 98 at my last corporate gig people will deploy wireless
network not knowing they reach beyond their office building or home. All of a sudden you have
access to the transmission media. With the wire you have to put a tap on it. Wireless gets
in places you didn't plan on. That's exactly what we have here. Another one, how many of
you ‑‑ this is my favorite. I'm not sure what to do about them. I play along but how many of you
got my call where it's like this is your bank center calling. Can you read me your 16 digit
number? I have. I'm worried there's fraud. Right? So I probably give it to them. Was
anyone here at the talk the gentlemen gave ‑‑ it deserves more recognition then I will
give it but they impersonated a cell tower and there are warnings outside the door. None
of you were here for that? Okay. The lights are bright. Okay. They don't call me eagle eyes.
So yeah. Great talk. And what was the key thing I took away was you don't have both parties
authenticating each other. You have the same thing here. With this set up E‑bay hardware you
can do a simpler method and you can become a node on their mesh. You just have to know the mesh
ID which can be read right off the air. That mesh ID. There's only 256 unique combinations
here. You read it off and you can join their network and participate. What can you do
with that? If you're a member of the mesh flower a mesh presumably because it's wireless
the health of the network is changing based on fog or all kinds of weather conditions or
magnetic transmissions. One node can report hey I have a great view to the back hall. You all
should route your traffic through me. A node on the network unlike a wired network
ruled to be in the right position. Not so with a mesh network. You can have the
traffic routed through you by issuing a favorable packet. Saying I have a fantastic link
to that last hop you need. I don't have to sit and sniff just with the cameras I'm in range
of. I can actually request all cameras. Or whatever other city services is present on that
network. All the equipment presumably also supports encryption of their own
transmission. Cameras, the audio transmitters, all those do. Never turned on. Like I said
using a directional antenna to secure the medium simply not sufficient. If I'm picking it up
while driving around a round something is wrong. Some of you may have worked with optical
transmitters where you shoot a laser to another ‑‑ to make a wireless link. You actually have
to get literally in between the two, the transmitter and receiver to pick it up. Not so
here. Antennas leak in directions you're not prepared for. I mentioned a fantastic
feature of Firetide. I really feel bad for the city and I'm putting the blame on the vendor.
Firetide actually supports mesh node authentication. PKI via certs, there's one node on the
mesh to all the other nodes. It's fantastic. Not turned on. That box didn't get checked
apparently. Again Automesh or any other protocol ‑‑ don't look at my typo. Simply not
sufficient. So really what we've seen a botched thing. Security is hard is it not? What do we
see? Is it actually broken? Certainly not. What was broken? Someone screwed up. Someone
thought managing it that way was a great idea and it wasn't. Someone thought doing something
a novel way instead of a proven way was a good idea and it wasn't. Someone here thought
they could ‑‑ because of other things in place that it was sufficiently secure or they
thought security wasn't a concern at all. We'll never know. So really this is really
the integrator. In some ways it's fun to knock your local cities or governments or what
not but really we put so much in the hands of private companies who take government money and
are supposed to provide a service. I presume the mayor pro tem of a small city like the one
we're doing up here that guy ‑‑ you have to feel bad for him. He's in no position to evaluate
the finished product even more than my grandmother is; should we knock her for saying oh honey
you have the whole internet on your phone. What do you expect from her? What do you expect
something from a city mayor or manager? They depend on the integrator. That's who they paid
substantial sums of money for. A near by city was paid half a million dollars for something
like a handful of cameras. It wasn't really related but it was a recent award or proposal, half
a million dollars. Will they get vendor security? It's 2014. I hope so. Is anyone in the city
capable of evaluating that install? I doubt it. The city of LA, city of Chicago, city of New
York. Sure. I would imagine so. Is there even an IT department. Is there anyone that can do a
post install test? One of the things I talk about in terms of providing IT service toss our
clients is do we test it in ways ‑‑ you know, here's something from me. Back in the
days of going desk to desk my last corporate gig when I set up a new computer for an employee I
forgot to install the printers probably because I don't print anything. I don't have a need to
print out paper. I never really have. But these people do. I don't understand. Our clients
print more than anybody. If that printer isn't there I will get a call immediately. In the same
way I feel like we often assume in the security world, you know, we ‑‑ here's another one.
Filtering services based on IP. Right? Certainly IP’s on the network. Did anyone actually
test it from an IP? Did anyone actually test it? Thank you. That sign was clear as day.
Thank you. I was nervous about missing that. Did anyone actually test it? I presume ‑‑
I'm guessing not. I think that's what I call the IT people, guilty here. If we do self
reflection we will see. We know what we're doing. We've done it a few times. A lot of times
maybe. It's worth checking. I know residents are upset that they're being monitored all the
time. I saw a comic recently that said how do you want to sell this to the American
people. Protecting children or fighting terrorists? They sold this in terms of public safety
and from there the public was calm enough, I'm guessing people wouldn't be happy if they found
out their sister's brother's kid could watch the cameras too or perhaps much worse going through
these potential threats beyond just security weakness. So here are some of the things we
theorized and tested. These things work. Big warning here. Federal and states are having
lawsuits that may apply. I'm not a lawyer or accountant. So seek the advice of someone that would
know about these things like the EFF. Don't go running afoul of wiretapping laws. So we've
talked about implementation like this one prior to 72 hours ago observing video streams is
trivial. Or because they're multicast you can subscribe yourself. Why not? Like with any
wireless system but especially one you can participate in the network in a meaningful way.
There's denial service opportunities. Simply by flooding, always subscribing.
Like I said these are normal off the shelf. Nothing special. They're not prepared to have a
thousand bogus subscriptions. ARP spoofing… You can become a node on the network or the
network video recorder. Dirty. Dirtyer though and totally worthy of a Doge is joining the
mesh legitimately. There's only 256ID's. All you have to do ‑‑ you can just read it and go oh
this is mesh number 3. Now as far as anyone can tell you're a legitimate node on their
network. I'm guessing that there aren't a lot of separation. There isn't a lot of separation
between these ‑‑ the cameras, the network DVR's and the rest of the city's networks. Often
they left themselves wide open to anyone who can listen or participate in the network with
accessing services that they're not prepared to have public access to. Honestly guys for the
police department you've got ‑‑ you have to start thinking about things like reliability. I mean
I'm not a lawyer but it would concern me potentially evidence could be tampered with from
50 miles away. >> You're basically plugged in. You might as well walk in. >> Find a hot
port in your office and you can participate in the network. Video manipulation. Think
Ocean's 11. You have the security guard with all the monitors and it's in the middle
of the night and he's watching and the guys you are supposed to root for takeover one of the
cameras and play back a loop. I call it all is well loop. An empty corridor. Whereas in
reality when the camera goes all the good guys are making their way down the corridor. Entirely
impossible. Videos usually is transmitted through packets. This is how it looks. This is my
illustration from this morning. You have a municipal camera connected to the municipal DVR.
These are packet numbers. Which are incremented as the next packet is sent. With a malicious
node, thank you, with a malicious node your own stream all we have to do is jump ahead.
So if the highest packet they're on is 106 I will jump to 200. My video packet is 200, 201, 202,
203. Think about what happens next. The DVR goes whoa, I’m getting ing packets out of
order. They've expired. I can ignore them. I've literally now just by incrementing the count
have taken over and the DVR has no reason not to, right? It's functioning as intended because
we're using EDP we can throw away all the packets ‑‑ now these packets will continue to
arrive and they're always ignored because somewhere along the line you watch the traffic
and see this happen. Packets arrive out of order. What do you do? You discard them. So just
with this, just by incrementing the UDP count you can completely takeover the network. Here is
the other danger. The cameras are trusted. I mean you can say hey let's have Godzilla walk
down the street. Or project all is well at the city bank but in reality it's being robbed. I'm
getting calls but I don't know Frank it looks fine from here. There's also ‑‑ you can also do
the opposite. Send resources else where through injecting false data like that. Not to
mention look I have a friend ‑‑ I don't want to identify him. I know someone that works at
verizon wireless. He tells me people look at all your dirty texts. Someone has access to
them all. Everyone can look at everyone's dirty pictures. Who do we trust to look at all this
data that's collected that's archived? These are privacy implications along with the
weaknesses that go along with it. Things that maybe not everyone thought deeply enough
about. Security is complicated and it interacts with other government things. I think some
times we think of ourselves as tech people and not citizens that have to participate in the
2014 world. So our demo is sad because of WEP. >> Because we can't break it. >> The person
who writes the check to us is now actually thinking because we ought not try because of
potential liability. I know we promised a demo and I'm very sorry. But thank you for coming
and filling this whole room. I think it's always a speaker's concern that, you know, when I
spoke two years ago there was a really, really, really great talk at the same time as mine
and I walked in and half the people bailed before I started. So thanks so much for your
attention. If anyone has questions we're going to be out front. [Applause]