Service Accounts suck - why data futures require end to end authentication.

 Can we all agree that "service" accounts suck from a security perspective.  Those are the accounts that you set up so what system/service can talk to another one.  Often this will be a database connection so the application uses one account (and thus one connection pool) to access the database.  These service accounts are sometimes unique to a service or application, but often its a standard service account for anything that needs to connect to a system.

The problem with that is that you've therefore got security defined at the service account level, not at all based on the users actually using it.  So if that database contains the personal information of every customer then you are relying on the application to ensure that they only display the information for a given customer, the security isn't with the data its with the application.

Back in 2003 a group called the "Jericho Forum" set up under Open Group to look at the infrastructural challenges of de-perimeterisation and they created a set of commandments the first of which is:

The scope and level of protection should be specific and appropriate to the asset at risk. 

Service accounts break this commandment as they take the most valuable asset (the data) and effectively remove security scope and place it in the application.   What needs to happen is that the original requestor of the information is authenticated at all levels, like with OAuth, so if I'm only allowed to see my data then if someone makes an error in the application code, or I run a Bobby Drop Tables attack, my "Select *" only returns my records.

This changes a lot of things, connection pooling for starters, but when you are looking at reporting in particular we have to get away from technologies that force systems accounts and therefore require multiple security models to be implemented within the consumption layer.

The appropriate level to protect data is at the data level and the scope is with the data only by shifting our perception of data from being about service accounts and databases to being about data being the asset can we start building security models that actually secure data as an asset.

Today most data technologies assume service accounts, this means that most data technologies don't think that data is an asset.  This has to change.

Getting RocksDB working on Raspberry PI (Unsatisfied linker error when trying to run Kafka Streams)

 If you are here its probably because you've tried to get RocksDB working on a Raspberry PI and had the following exception:

Exception in thread "main-broker-b066f428-2e48-4d73-91cd-aab782bd9c4c-StreamThread-1" java.lang.UnsatisfiedLinkError: /tmp/librocksdbjni7453541812184957798.so: /tmp/librocksdbjni7453541812184957798.so: cannot open shared object file: No such file or directory (Possible cause: can't load IA 32 .so on a ARM platform)

at java.base/java.lang.ClassLoader$NativeLibrary.load0(Native Method)

at java.base/java.lang.ClassLoader$NativeLibrary.load(ClassLoader.java:2452)

The reason for this is that your rocksdbjni jar file doesn't include the required shared library (.so file) for the ARM platform, it doesn't include x86 for Linux, a Windows DLL and a PPC(!) library but not for Linux so you are going to have to roll your own.

Step 1 - Doing the apt-gets

You have to do the build ON the Raspberry PI as its C code that needs to be compiled and the instructions for rocks aren't overly helpful as they assume a couple of things are already installed.

The following appear to be required:

sudo apt-get install cmake

sudo apt-get install vagrant

sudo apt-get update --fix-missing

sudo apt-get install vagrant

The vagrant part appears to be not actually required but I had issues when it wasn't installed so it is probably a dependency of the apt-get pulls in.

Step 2 - getting the code for RocksDB

Now you are going to have to download the code for RocksDB, I hope you've got a fairly large SD card as this will take up around a gig of space all told

Make a directory under your home directory (I call mine 'dev') and change into the directory (cd dev).

git clone https://github.com/facebook/rocksdb.git rocksdb

Then change directory into rocksdb.  If you are a Java developer you are possibly about to discover for the first time a tool that is why Java people created ant and maven because they REALLY hated how it worked...

Step 3 - running make

Then you want to run 

make rocksdbjavastaticrelease

Which will take a while but produce you a nice JNI jar file for your Raspberry PI that you can add to your classpath and you will be away.... the jar file will be under the "target" directory under rocksdb, so if you are on the default install with user "pi" it will be 

/home/pi/dev/rocksdb/java/target/rocksdbjni-6.12.0-linux32.jar

Copy that file into your classpath and rocks DB should now work fine

Step 4 - if running a Kafka Streams application

I'm sure there is a logical reason for this and that I could have created another work around, but having run the original and had the linker error when it accessed RocksDB there was a new error when running Kafka Streams:

Exception in thread "main-broker-00a490b4-b50a-4cc4-aded-d6324ad0f291-StreamThread-1" org.apache.kafka.streams.errors.ProcessorStateException: Error opening store reading-topic-STATE-STORE-0000000000 at location /tmp/kafka-streams/main-broker/0_0/rocksdb/reading-topic-STATE-STORE-0000000000

at org.apache.kafka.streams.state.internals.RocksDBTimestampedStore.openRocksDB(RocksDBTimestampedStore.java:87)

at org.apache.kafka.streams.state.internals.RocksDBStore.openDB(RocksDBStore.java:188)

at org.apache.kafka.streams.state.internals.RocksDBStore.init(RocksDBStore.java:224)

at org.apache.kafka.streams.state.internals.WrappedStateStore.init(WrappedStateStore.java:48)

at org.apache.kafka.streams.state.internals.ChangeLoggingKeyValueBytesStore.init(ChangeLoggingKeyValueBytesStore.java:42)

This is weird as it had never successfully run (as it couldn't create the DB), but for some reason there is state somewhere (I assume in the broker) that still exists for the app id, I assume because that runs before the linker.  So the fix there is to change "application.id" in the kafka configuration file to be something new and it should all run fine.

Why taking good holidays is good practice

Back when I was a fairly recent graduate I received one of the best pieces of advice I've ever received.  The project was having some delivery pressures and I was seen as crucial to one of the key parts.  As a result my manager was putting pressure on me to cancel my holiday (two weeks of Windsurfing bliss in the Med with friends) with a promise that the company would cover the costs.  I was called into the BIG boss' office with the full expectation that he would put the screws on further and in my head I had a list of additional demands.

"Steve, I've heard that XXX [name redacted] wants you to cancel your holiday"
"Yes, we've got to get the release out and I'm the one who knows most about resolving issues in the team"
"Don't"

This kind of stumped me so I sat there a bit quiet, and here came the best advice ever

"If you cancel this holiday then all anyone will remember is that they can make you cancel holidays, they won't actually appreciate it.  If you go on holiday and it all collapses all everyone will remember is that it collapsed without you, if it goes badly or much harder because you aren't there they will remember that, and if its OK without you then hopefully you have enough talent to not insist that the universe revolves around you.  Go on holiday, don't be contactable and remember that now more senior folks know about you because you didn't cancel your holiday."

Its a mantra I've lived by.  Take your holidays, plan to take them, and take them PROPERLY.  That means

1. Change your password before leaving and DON'T update it on any mobile devices
2. Your work laptop does NOT travel on holiday with you
3. You don't do conference calls, standups, "just 30 minutes a day" or any other nonsense.

Since I've become a manager I think this is more important.  If I can't manage and succession plan to the level that the universe doesn't collapse while I'm away then what sort of manager am I?  I actually look very badly on folks who don't take holidays properly as they risking promoting a "macho hero" culture rather than a decent work/life balance culture.  If you don't recharge your batteries properly, spend proper time with your family then really, what is the point?

Planning for holidays is a sign of good planning, requiring people to cancel holidays is a sign of bad planning.  Requiring yourself to cancel holidays is a sign of extremely bad planning and bad succession management.

If you are so bad a manager that your team can't cope for 2 weeks without you, then you need to look at how you are developing your next level.  If you are so scared that people won't "miss" you when you are out of the office then you really need to check yourself and look at your career ambitions and direction.

Taking a good holiday (vacation for my American colleagues) is one of the key reasons WHY we work, explore the world, meet new people, do new things, RELAX. 

Back to the story at the top, I came back after the holiday to see issues lists and problems, it took 4 days of hard work to get us back to level and everyone saw me as the hero.  Had I cancelled my holiday none of the management team would have known how crucial I was to the team's success, because of the holiday I was rapidly promoted into a new role.  As a developer taking that holiday led directly to people appreciating much more what I did for the team.

These days a key success factor for my team is how they cope brilliantly when I'm not there, I'm not worried that this means I'm irrelevant, it means that as we grow and take on new challenges my leadership team is able to grow and develop and take those challenges on, leaving me free to work on what is next.

Good holidays are good practice.

The ten commandments of IT projects

And lo a new project did start and there was much wailing and gnashing of teeth, for up on the board had been nailed ten commandments that the project must follow and the developers were sore afraid.

1. Thou shalt put everything in version control, yeah even the meeting minutes, presentations and "requirements documents that aren't even finished yet" for without control everything is chaos
2. Thou shalt not break the build
3. Thou shalt never deploy to a shared environment anything that is not under version control
4. Thou shalt be honest about how long a task will take
5. Thou shalt test both the happy and unhappy path
6. Thou shalt automate everything that can be automated and do so early
7. Thou shalt not re-invent the wheel "just because"
• Most specifically thou art not Doug Lea, do not create your own threading library
• If you are Doug Lea then just use your own threading library
8. Assume Murphy's Law to be always true and never state "that is very unlikely to happen"
9. Code unto others as you would have them code towards you
10. Do not screw with the environment configurations without first verifying you are not messing things up for others

The punishment for breaking these commandments is to be relegated to the project management office and be responsible for the many and various Excel spreadsheets report that people outside of the project appear to hold sacred.  

-----

Now because I know that folks will look to interpret these things in many ways lets be clear on a few.

9. Code unto others as you would have them code towards you

This is important, you should write code that is designed to be understood by someone else and written in a way that if you came to it fresh you would be able to understand it.

6. Thou shalt automate everything that can be automated and do so early

This is important, too often people don't automate and more importantly don't automate early, trying to retrofit the automation once the project reaches a certain size.  You should really have everything automated before a single line of code is written, something that 'does nothing successfully'

5. Thou shalt test both the happy and the unhappy path 

8. Assume Murphy's Law is true

Too often I've heard the phrase "that won't happen" followed by "it happened in the first day in production".  This is why its really important to test the unhappy path and to clearly identify the bounds you are going to be operating within.  Hoping that something won't happen isn't a strategy.

Making DevOps Business Driven - a service view

I've been doing a bit recently around DevOps and what I've been seeing is that companies that having been scaling DevOps tend to run into a problem: exactly what is a good boundary for a DevOps team? Now I've talked before about how Microservices are just SOA with a new logo, well there is an interesting piece about DevOps as well, its not actually a brand new thing.  Its an evolution and industrialisation of what was leading practice several years ago.

Back in 2007 I gave a presentation on why SOA was a business challenge (full deck at the end) and in there were two pictures that talked about how you needed to change the way you thought about services:

So on the left we've got a view that says that you need to think about a full lifecycle, and on the right you've got a picture that talks about the needs to have an architect, owner and delivery manager (programme manager)
This is what we were doing around SOA projects back in 2007 as a structure and getting the architects and developers (but ESPECIALLY the architects) to be accountable for the full lifecycle.  Its absolutely fantastic to see this becoming normal practice and there are some great lessons out there and technical approaches.

One thing I've not seen however is an answer to what my DevOps team is and how I manage a large number of DevOps teams.  This is where Business Architecture comes in, the point here is that its not enough to just have lots and lots of DevOps teams, you need to align those to the business owners and align them to the structure that is driving them.  You also need to have that structure so one team doesn't just call the 'Buy from Ferrari' internal service without going through procurement first for approval.

So in a DevOps world we are beginning to realize the full-lifecycle view on Business Services, providing a technical approach to automating and managing services that look like the business, evolve like the business and provide the business a structure where they can focus costs where it delivers the most value.

There is much new in the DevOps world, but there is also much we can learn from the Business Architecture space on how to set up DevOps teams to better align to the business and enable DevOps to scale at traditional complex organisations as well as more simple (from a business model perspective) internet companies.

Big Data and the importance of Meta-Data

Data isn't really respected in businesses, you can see that because unlike other corporate assets there is rarely a decent corporate catalog that shows what exists and who has it.  In the vast majority of companies there is more effort and automation put into tracking laptops than there is into cataloging and curating information.

Historically we've sort of been able to get away with this because information has resided in disparate systems and even those which join it together, an EDW for instance, have only had a limited number of sources and have viewed the information only in a single way (the final schema).  So basically we've relied on local knowledge of the information to get by.  This really doesn't work in a Big Data world.

The whole point in a Big Data world is having access to everything, being able to combine information from multiple places within a single Business Data Lake so you can allow the business to create their own views.

Quite simply without Meta-Data you are not giving them any sort of map to find the information they need and help them understand the security required.  Meta-Data needs to be a day one consideration on a Big Data program, by the time you've got a few dozen sources imported its going to be a pain going back and adding the information.  This also means the tool used to search the Meta-Data is going to be important.

In a Big Data world Meta-Data is crucial to make the Data Lake business friendly and essential in ensuring the data can be secured.    Lets be clear here HCatalog does matter but its not sufficient, you can do a lot with HCatalog but that is only the start because you've got to look about where information comes from, what its security policy is, where you've distilled that information to.  So its not just about what is in the HDFS repository its about what you've distilled into SQL or Data Science views, its about how the business can access that information not just "you can find it here in HDFS".

This is what Gartner were talking about in the Data Lake Fallacy but as I've written elsewhere, that sort of missed the point that HDFS isn't the only part of a data lake and EDW approaches only solve one set of problems not the broader challenge of Big Data.

Meta-Data tools are out there, and you've probably not really looked at them but here is what you need to test (not a complete list, but these for me are the must have requirements).

1. Lineage from source - can it automatically link to the loading processes to say where information came from?
2. Search - Can I search to find the information I want?  Can a non-technical user search?
3. Multiple destinations - can it support HDFS, SQL and analytical destinations
4. Lineage to destination - can it link to the distillation process and automatically provide lineage to destination
5. Business View - can I model the business context of the information (Business Service Architecture style)
6. My own attributes - can I extend the Meta-data model with my own views on what is required?

The point of modelling in a business context is really important.  Knowing information came from an SAP system is technically interesting, but knowing its Procurement data that is blessed & created by the procurement department (as opposed to being a secondary source) is significantly more valuable.  If you can't present the meta-data in a business structure you aren't going to get the business users able to use it, its just another IT centric tool.

The advantage of Business Service structured meta-data is that it matches up to how you evolve and manage your transactional systems as well.

Security Big Data - Part 7 - a summary

Over six parts I've gone through a bit of a journey on what Big Data Security is all about.

1. Securing Big Data is about layers
2. Use the power of Big Data to secure Big Data
3. How maths and machine learning helps
4. Why its how you alert that matters
5. Why Information Security is part of Information Governance
6. Classifying Risk and the importance of Meta-Data

The fundamental point here is that encryption and ACLs provide only a basic hygiene factor when it comes to securing Big Data.  The risk and value of information is increasing and by creating Big Data solutions businesses are creating more valuable and therefore more at risk information solutions.  This means that Information Security needs to become a fundamental part of Information Governance and that new ways of securing that information are required.

This is where Big Data comes to its own rescue through the use of large data sets which enable new generations of algorithms to identify and then alert based on the risk and the right way to handle it.  This all requires you to consider Information Security as a core part of the Meta-data that is captured and governed around information.

The time to start thinking, planning and acting on Information Security is now, its not when you become the next Target or when one of your employees becomes your own personal Edward Snowden, its now and its about having a business practice and approach that considers information as a valuable asset and secures it in the same way as other assets in a business are secured. 

Big Data Security is a new generation of challenges, and a new generation of risks, these require a new generation of solutions and a new corporate culture where information security isn't just left to a few people in the IT department.

Securing Big Data Part 6 - Classifying risk

So now your Information Governance groups consider Information Security to be important you have to then think about how they should be classifying the risk.  Now there are docs out there on some of these which talk about frameworks.  British Columbia's government has one for instance that talks about High, Medium and Low risk, but for me that really misses the point and over simplifies the problem which ends up complicating implementation and operational decisions.

In a Big Data world its not simply about the risk of an individual piece of information, its about the risk in context.  So the first stage of classification is "what is the risk of this information on its own?" its that sort of classification that the BC Government framework helps you with.  There are some pieces of information (The Australian Tax File Number for instance) where their corporate risk is high just as an individual piece of information.  The Australian TFN has special handling rules and significant fines if handled incorrectly.  This means its well beyond "Personal Identification Information" which many companies consider to be the highest level.  So at this level I'd recommend having Five risk statuses

1. Special Risk - Specific legislation and fines apply to this piece of information
2. High - losing this information has corporate reputation and financial risk
3. Medium - losing this information can impact corporate competitiveness
4. Low - losing this information has no corporate risk
5. Public - the information is already public

The point here is that this is about information as a single entity, a personal address, a business registration, etc.  That is only the first stage when considering risk.

The next stage is considering the Direct Aggregation Risk this is about what happens when you combine two pieces of information together, do that change the risk.  The categories remain the same but here we are looking at other elements.  So for instance address information would be low risk or public, but when combined with a person that link becomes higher risk.  When looking at corporate information on sales that might be medium risk, but when that is tied to specific companies or revenue it could become a bigger risk.  Also at this stage you need to look at the policy of allowing information to be combined and you don't want to have a "always no" policy.

So what if someone wants to combine personal information with twitter information to get personal preferences?  Is that allowed?  What is the policy for getting approval for new aggregations, how quickly is risk assessed and is business work allowed to continue while the risk is assessed? When looking at Direct Aggregation you are often looking at where the new value will come from in Big Data so you cannot just prevent that value being created.  So setting up clear boundaries of where approval is required (combining PII information with new sources requires approval for instance) and where you can get approval after the fact (sales data with anything is ok, we'll approve at the next quarterly meeting or modify policy).

The final stage is the most complex its the Indirect Aggregation Risk that is the risk of where two sets of aggregated results are combined and though independently they are not high risk the pulling together of that information constitutes a higher level risk.  The answer to this is actually to simplify the problem and consider aggregations not as just aggregations but as information sources in their own right. 

This brings us to the final challenge in all this classification: Where do you record the risk?

Well this is just meta-data, but that is often the area that companies spend the least amount of time thinking about but when looking at massive amounts of data and particularly disparate data sources and their results then Meta-Data becomes key to big data.  But lets look just at the security side at the moment.

Data	Type	Direct Risk
Customer	Collection	Medium
Tax File Number	Field	Special
Twitter Feed	Collection	Public

and for Aggregations

Source 1	Source 2	Source 3	Source 4	Aggregation Name	Aggregation Risk
Customer	Address	Invoice	Payments	Outstanding Consumer Debt	High
Customer	Twitter	Locaiton		Customer Locations	Medium
Organization	Address	Invoice	Payments	Outstanding Company Debt	Low

The point here is that you really need to start thinking about how you automate this, what tools you need.  In a Big Data world the heart of security is about being able to classify the risk and having that inform the Big Data anomaly detection so you can inform the right people and drive the risk.

This gives us the next piece of classification that is required which is about understanding who gets informed when there is an information breach.  This is a core part of the Information Governance and classification approach, because its hear that the business needs to say "I'm interested when that specific risk is triggered".  This is another piece of Meta-data and one that then informs the Big Data security algorithms who should be alerted.

If classification isn't part of your Information Governance group, or indeed you don't even have a business centric IG group then you really don't consider either information or its security to be important.

Other Parts in the series

1. Securing Big Data is about layers
2. Use the power of Big Data to secure Big Data
3. How maths and machine learning helps
4. Why its how you alert that matters
5. Why Information Security is part of Information Governance

Securing Big Data Part 5 - your Big Data Security team

What does your security team look like today?

Or the IT equivalent, "the folks that say no".  The point is that in most companies information security isn't actually something that is considered important.  How do I know this?  Well because basically most IT Security teams are the equivalent of the nightclub bouncers, they aren't the people who own the club, they aren't as important as the barman, certainly not as important as the DJ and in terms of Nightclub strategy their only input will be on the ropes being set up outside the club.

If Information is actually important then information security is much more than a bunch of bouncers trying to keep undesirables out.  Its about the practice of information security and the education of information security,  in this Information security is actually a core part of Information Governance and Information Governance is very much a business led thing.

Big Data increases the risks of information loss, because fundamentally you are not only storing more information you are centralizing more information which means more inferences can be made, more links made and more data stolen.  This means that historical thefts which stole data from a small number of systems risk being dwarfed by Big Data hacks which steal huge sets or even runs algorithms within a data lake and steals the results.

So when looking at Big Data security you need to split into three core groups

The point here is that this governance is exactly the same as your normal Data governance, its essential that Information Security becomes a foundation element of information governance.  The three different parts of governance are set up because there are different focuses

1. Standards - sets the gold standard of what should be achieved
2. Policy - sets what can be achieved right now (which may not meet the gold standard)
3. KPI Management - tracks compliance to the gold standard and adherence to policy

The reason these are not just a single group is that the motivations are different.  Standards groups set up what would be ideal, its against this ideal that progress can be tracked.  If you combine Standards groups with Policy groups you end up with Standards which are 'the best we can do right now' which doesn't give you something to track towards over multiple years.

KPI management is there to keep people honest.  This is the same sort of model I talked about around SOA Governance and its the same sort of model that whole countries use, so it tends to surprise me when people don't understand the importance of standards v policy and the importance of tracking and judging compliance independently from those executing.

So your Big Data Security team starts and ends with the Information Governance team, if information security isn't a key focus for that team then you aren't considering information as important and you aren't worried about information security.

Other Parts in the series

1. Securing Big Data is about layers
2. Use the power of Big Data to secure Big Data
3. How maths and machine learning helps
4. Why its how you alert that matters

Securing Big Data - Part 4 - Not crying Wolf.

In the first three parts of this I talked about how Securing Big Data is about layers, and then about how you need to use the power of Big Data to secure Big Data, then how maths and machine learning helps to identify what is reasonable and was is anomalous.

The Target Credit Card hack highlights this problem.  Alerts were made, lights did flash.  The problem was that so many lights flashed and so many alarms normally went off that people didn't know how to separate the important from the noise.  This is where many complex analytics approaches have historically failed: they've not shown people what to do.

If you want a great example of IT's normal approach to this problem then the ethernet port is a good example.

What does the colour yellow mean normally? Its a warning colour, so something that flashes yellow would be bad right?  Nope it just means that a packet has been detected... err but doesn't the green light already mean that its connected?  Well yes but that isn't the point, if you are looking at a specific problem then the yellow NOT flashing is really an issue... so yellow flashing is good, yellow NOT flashing is bad...

Doesn't really make sense does it?  Its not a natural way to alert.  There are good technical reasons to do it that way (its easier technically) but that doesn't actually help people.

With security this problem becomes amplified and is often made worse through centralising reactions to a security team which knows security but doesn't know the business context.  The challenge therefore is to categorize the type of issue and have different mechanisms for each one.  Broadly these risks split into 4 groups

Its important when looking at risks around Big Data to understand what group a risk falls into which then indicates the right way to alert.  Its also important to recognize that as information becomes available an incident may escalate between groups.

So lets take an example.  A router indicates that its receiving strange external traffic.  This is an IT operations problem and it needs to be handled by the group in IT ops which deals with router traffic.  Then the Big Data security detection algorithms link that router issue to the access of sales information from the CRM system.  This escalates the problem to the LoB level, its now a business challenge and the question becomes a business decision on how to cut or limit access.  The Sales Director may choose to cut all access to the CRM system rather than risk losing the information, or may consider it to be a minor business risk when lined up against closing the current quarter.  The point is that the information is presented in a business context, highlighting the information at risk so a business decision can be taken.

Now lets suppose that the Big Data algorithms link the router traffic to a broader set of attacks on the internal network, a snooping hack, this is where the Chief Information Security Officer comes in, that person needs to decide how to handle this broad ranging IT attack, do they shut down the routers and cut the company off from the world?  Do they start dropping and patching, and do they alert law enforcement.

Finally the Big Data algorithms find that credit card data is at risk, suddenly this becomes a corporate reputation risk issue and needs to go to the Chief Risk Officer (or the CFO if they have that role) to take the pretty dramatic decisions that need to be made when a major cyber attack is underway.

The point here though is that it needs to be systematic how its highlighted and escalated, it can't all go through a central team.  The CRO needs to be automatically informed when the risk is sufficient, but only be informed then.  If its a significant IT risk then its the job of the CISO to inform the CRO, not for every single risk to be highlighted to the CRO as if they need to deal with them.

The basic rule is simple: "Does the person seeing this alert care about this issue? Does the person seeing this alert have the authority to do something about this issue? and finally: does the person seeing this alert have someone lower in their reporting chain who answers 'yes' to those questions?"

If you answer "Yes, Yes, No" then you've found the right level and then need to concentrate on the mechanism.  If its "Yes, Yes, Yes" then you are in fact cluttering if you show them everything that every person in their reporting tree handles as part of their job.

In terms of the mechanism its important to think on that "flashing yellow light" on the Ethernet port.  If something is ok then "Green is good", if its an administrative issue (patch level on a router) then it needs to be flagged into the tasks to be done.  If its an active and live issue it needs to come front and center.

In terms of your effort when securing Big Data you should be putting more effort into how you react than on almost any other stage in the chain.  If you get the last part wrong then you lose all the value of the former stages.  This means you need to look at how people work, look at what mechanisms they use, so should the CRO be alerted via a website they have to go to or via an SMS to the mobile they carry around all the time and that take them to a mobile application on that same device? (hint: its not the former).

This is the area where I see the least effort made and the most mistakes being made, mistakes that are normally "Crying Wolf" so you show every single thing and expect people to filter out thousands of minor issues and magically find the things that matter.

Target showed that this doesn't work.

Securing Big Data - Part 3 - Security through Maths

In the first two parts of this I talked about how Securing Big Data is about layers, and then about how you need to use the power of Big Data to secure Big Data.  The next part is "what do you do with all that data?".   This is where Machine Learning and Mathematics comes in, in other words its about how you use Big Data analytics to secure Big Data.

What you want to do is build up a picture of what represents reasonable behaviour, that is why you want all of that history and range of information.  Its the full set of that across not single actions but millions of actions and interactions that builds the picture of reasonable.  Its reasonable for a sys-admin to access a system, its not reasonable for them to download classified information to a USB stick.

A single request is something you control using an ACL, but that doesn't include the context of the request (its 11pm, why is someone accessing that information at all that late?).

You also need to look at the aggregated requests - They've looked at the next quarters sales forecast while also browsing external job hunting sites and typing up a resignation letter.

Then you need to look at the history of that - Oh its normal for someone to be doing that at quarter end, all the sales people tend to do that.

This gives us the behaviour model for those requests which leads to us understanding what is considered reasonable.  From reasonable we can then identify anomalous behaviour (behaviour that isn't reasonable).

No human defined and managed system can handle this amount of information, but Machine Learning algorithms just chomp up this sort of data and create the models for you.  This isn't a trivial task and its certainly massively more complex than the sorts of ACLs, encryption criteria and basic security policies that IT is used to.  These algorithms need tending, they need tuning and they need monitoring.

Choosing the right type of algorithms (and there are LOTS of different choices) is where Data Scientists come in, they can not only select the right type of algorithm but also tune and tend it so it produces the most effective set of results consistently.

What this gives you however is business centric security, that is security that looks at how a business operates.  Anomalous Behaviour Detection therefore represents the way to secure Big Data by using Big Data.

The final challenge is then on how to alert people so they actually react.

Securing Big Data - Part 2 - understanding the data required to secure it

In the first part of Securing Big Data I talked about the two different types of security.  The traditional IT and ACL security that needs to be done to match traditional solutions with an RDBMS but that is pretty much where those systems stop in terms of security which means they don't address the real threats out there, which are to do with cyber attacks and social engineering.  An ACL is only any good if people do what they are expected to do.  Both the Target Credit Card hack and the NSA hack by Edward Snowden involved some sort of insider breach, so ACL approaches were part of the problem not the solution.

With Big Data and Hadoop we have another option: to actually use Hadoop to secure the information in Hadoop.  This means getting more information into Hadoop, particularly network and machine log information.

So why do you need all this information?  Well you need obviously to know the access requests and what data is accessed, but you also need to know where the network packets go.  Are they going to a normal desktop or corporate laptop or do they head out of the company and onto the internet?  Does the machine that is downloading information have a USB drive plugged in and the files being copied to it?  Is there a person logged into the machine or is it just a headless workstation?

The point here is that when looking at securing Big Data you need to take a Big Data view of security.  This means going well beyond traditional RDBMS approaches and not building a separate security silo but instead looking at the information, adding in information about how it is accessed and information about where that is accessed from and how.

This information builds up a single request view, but by storing that information you can start building up a profile of information to understand what other information has been requested and how that matches or can be linked.  Thus if someone makes 100 individual requests that on their own are OK but taken in aggregate represent a threat then its the storage of that history of requests that give you the security.

So to secure Big Data we don't need to just look at individual requests, the ACL model, we need to start building up the broad picture of how the information is accessed, what context it is accessed within and where that information ends up.  To secure Big Data, or in reality any data, is about looking at the business context rather than the technical focus of encryption and ACLs which are simply hygiene factors. 

Securing Big Data - Part 1

As Big Data and its technologies such as Hadoop head deeper into the enterprise so questions around compliance and security rear their heads.

The first interesting point in this is that it shows the approach to security that many of the Silicon Valley companies that use Hadoop at scale have taken, namely pretty little really.  It isn't that protecting information has been seen as a massively important thing as there just aren't the basic pieces within Hadoop to do that.  Its not something that has been designed to be secure from day one, its designed to be easy to use and to do a job.  Its as governments and enterprises with compliance departments begin to look at Hadoop that these requirements are really surfacing.

But I'm not going to talk about the encryption and tokenization solutions, those are hygiene factors but not really about securing the information, because its still going to be accessible to people with the right permissions.  Because of that its still at risk from Cyber or Social Engineering attacks.  The means that security for Big Data is about layering and really its about two different ways of viewing security.

IT solutions, technical solutions, can really help around access control, encryption but they don't really help you to actually prevent information being stolen.  What stops information leaking is about the business decisions you make.

The first one of those is: what information do I actually store?  So you might decide that you'll just store IDs for customer information in Hadoop and then use a more traditional store to provide the cross reference of IDs back to customer information when its required.

Above this however is actually using Hadoop to protect Hadoop.  What is Hadoop?  Its a Big Data analytics platform.  What is the biggest threat around Data?  Social Engineering or Cyber 'inside the walls' attacks.  So the first stage in securing Hadoop is to do the IT basics, but the next stage of securing the business of information is about using the power of Hadoop to secure the information stored within it.

Uber won't want drivers in the future

I'm an Uber user, its a great service outside of cities with decent public transport.  But I have been thinking about where they will justify the $17bn valuation and give people a return on that $1.2bn investment.  At the same time I've been following the autonomous car pieces with interest and I think there is a pretty clear way this can end, especially as Uber have already said they are going to buy 2,500 Google cars.

If Uber do push local taxi firms to the wall leveraging that huge cash pile then it gives them a great 'front-window' in terms of the services they offer.  Now a the moment part of the beauty of the model is that they don't need to pay benefits to the drivers.  But what if you could get rid of the drivers?  Suddenly it made me think of the net impact of this.  Firstly with a driverless bus you could look at ad-hoc public transport style pricing.  Change the app so you put in where you are and where you want to go to and Uber can start ride-sharing to maximise utilisation of larger vehicles.  This reduces the cost to the consumer and also increases the profitability.  It also would put Uber into direct competition with the public transport systems in many countries, again leveraging that $1.2bn war chest to initially under-price and building on the political climates in some countries that see private transportation as the best way to go.

This all leaves Uber drivers being squeezed out.  Right now Uber take a 20% cut, but don't have the risk around the driver and the car.  But with electric autonomous vehicles much of that risk is going to be reduced which enables Uber to start offering UberAuto where they use some of that war chest to buy electric autonomous vehicles and have them compete in the UberX type of market, potentially directly with Uber X drivers or at a lower price as Uber could take the full revenue from the fare.  A franchise model where people buy 'official' Uber autonomous vehicles and have them added to the pool would spread the risk but essentially change the model away from human drivers.

To get this position of strength though what Uber really needs today is to push taxi firms out of the way so they are see as the obvious first place to get a ride in a city.  They are already doing that with their pricing, and having that $1.2bn means they are far from being a little startup company.  They are a leveraged market approach where the war chest gives them the ability to out compete local competitors.  Uber are doing this by leveraging cheap (to them) labour from the 'shared economy', or in reality are creating a business based around zero hours contracts and no benefits.

Longer term though its hard to see why Uber would keep sticking with people as autonomous vehicles become more cost effective.  By cutting out drivers they reduce a degree of risk and also increase the share of revenue that they can take.  Uber are clearly thinking this way with the purchase of the Google cars and its something we can expect to see increase over time.  This shift will bring Uber into the local public transport market where it can provide more flexible routing (at a price) than traditional bus and train services, but still provide a degree of cost sharing.  The 101 in San Francisco would be a great example of a profitable 'bus' route for Uber where people are dropped and picked up from home and work but where 80%+ of the journey is shared.

When looking at Uber lots of people talk about the 'sharing economy' but my prediction is the future of Uber is as an autonomous vehicle fleet, the sharing economy (and that war chest) just positions it ready for that future and helps remove some of the competition so its a clearer market when the shift happens.

Whistler, Microsoft and how far cloud has come

In six years Microsoft has come from almost zero corporate knowledge about how cloud computing works to it being an integral part of their strategy.  Sure back in early 2008 there were some pieces of Microsoft that knew about cloud but that really wasn't a corporate view it was what a very few people inside the company knew.

How do I know this? Well back in 2008 I was sitting on the top of a mountain with Simon Plant in Whistler.  The snow wasn't great that season but there are few places that I'd rather take a conference call.  The conference call was with Microsoft's licensing folks discussing how we can license their technology, SQL Server, Sharepoint etc on AWS.  It was a rather interesting conversation to say the least.

We were asking how they'd license for virtual machines, and how things like license portability worked in virtual environments.  A typical exchange would go something like

Simon: "So what we need is a virtual core price"
MSFT: "That will be the same as the physical core price"
Simon: "But its ok that it might move physical machines?"
MSFT: "As long as its less than once every 90 days yes"
Simon: "It could be more than that"
Me: "It could be every hour"
MSFT: "No problems, you'll just need to license every core it goes on"
Me: "We don't know what physical cores it runs on"
MSFT: "Why not?"
Simon: "Because its a cloud platform, we don't care about the physical boxes"

Then the conversation included one of the finest lines to ever come out of a software companies mouth

"Well to be safe you just need to ask Amazon how many cores they have in the Data Centre and license for that"

The reason I re-tell this story is to make the point at just how far we've come in 6 years.  I don't think any licensing person would suggest today that you'd need to license for every physical core in an entire data centre.  There really wasn't an understanding that we couldn't just ask Amazon for its core count in every data centre or that we didn't even know physically lived, the bit they really couldn't get was the idea that we didn't care and not knowing those things was actually a positive.

The call continued and by the end we were actually getting somewhere with a general acceptance that physical to virtual licensing needed some wording changes to get it working on AWS.  The Microsoft guys were pretty receptive and keen to learn but it was clearly a new set of concepts for them.

Then Mr Plant blew their mind

Simon: "What about scale down?"
MSFT: "What do you mean?"
Simon: "Well the point of cloud is to scale up and down, so what do we do when we scale down?"
MSFT: "You just need to license at peak usage"
Me: "But that destroys the whole idea of dynamic scaling"
MSFT: "Why?"
Simon: "Well if you scale once a year for a peak for a couple days, say for financial reporting, the rest of the year that just remains idle which is wasted money"

The concept of temporary licenses and dynamic scaling was clearly one that went way beyond what they were able to do at that stage.  There were more conversations then explaining about what cloud really meant and the sorts of things customers would be asking them for in years to come.  This whole call took place with Simon at 12,000ft and him about 12 feet further up the mountain so we wouldn't get interference.  The Microsoft team commented that we appeared very co-ordinated given we were dialing in from UK and US numbers and we just didn't think saying 'actually we are sitting with snowboards on our feet' was terribly professional.

The above conversation was repeated with pretty much every single software vendor over about 3 months with the same misunderstanding and same suggestions of 'license the whole data centre', I'm just singling out the Microsoft example as they are probably now one of the biggest proponents of cloud and it sits at the core of their strategy... oh and doing a conference call on a snowboard was cool.

Six years is what its taken to go from there to here, a world where cloud is now practically the default approach, whether public, private or hybrid and those questioning cloud are effectively the uneducated minority, just as Microsoft were back in 2008.  Now the challenge for enterprises is understanding just how they take on these challenges at enterprise scale, and that is what Simon has been doing since then, leading to him setting up his own business Dual Spark which specialises in exactly that.

Simon Plant: doing cloud computing for longer than Microsoft.

Big Data doom mongers need to look outside of the marketing department

In every change there are hype machines that over play and sages who call doom.  Into the Big Data arena steps David Searls to proclaim that Big Data is a myth and simply hype which is set to burst in an article over at ZDNet.

But big data, he said, is nothing more than the myth that collecting vast amounts of data can help companies know customers better than those customers even know themselves.

 The boogie men in this story are IBM and the consultants who have hyped it all up.  There another sage jumps in

Dr Matthew Landauer, co-founder of OpenAustralia, is equally sceptical about big data. "All it allows you to do is optimise your current business," he told ZDNet. "It's never going to tell you that you're doing business wrong or need another model.

There then moves forward a complaint about privacy and security (which I'm not disputing) but the key point is that Big Data is a bubble, and I really have to disagree with the definition of big data, the lack of innovation it can drive and that its a bubble.

Firstly I don't agree that marketing and customer data is what Big Data is about.  The vast majority of my conversations on Big Data have nothing to do with an explosion of customer information (e.g social media) but are instead about machine data, trading data, weather data and other massive data sets that historically companies couldn't cost effectively do analytics on.

Social Media and customer information is just one part of the challenge and its probably the most fluffy bunny and liable to be a bubble but to infer from there that Big Data is just hype is like assuming all swans are white because you see a single white swan.

Secondly is the assertion is that it cannot tell you that you are doing things incorrectly.  I'm not sure what sort of machine learning and other data science work Matthew Landauer has done but I am surprised that someone with a PhD in Physics from Cambridge hasn't seen examples from his own field at just how much change becoming data driven can deliver in terms of insight that causes disciplines,  companies and industries to make dramatic changes and have new approaches (the LHR at CERN for instance is quite clearly a Big Data application).  Finance is covered with examples where smarter algorithms identified that things were done wrong and that new ways would make more money.  By analysing and simulating you can absolutely find that there are new ways that can work significantly better.

Thirdly I disagree about who started the big data hype, IBM were far from being the leaders, that job goes to two industries.  Firstly the internet giants, Amazon, Google, eBay, Yahoo etc who created entire new business models based on information, and secondly on engineering companies who saw new business models based on information.  Sure the 'marketing/social media' has come to be the default story used by the lazy but that is far from saying that it is actually the story.

Big Data marketing might pop, that doesn't mean that Big Data is hype.  Saying so would be like claiming the failure of Association Football to become the dominant sport in North America means that Association Football is failing.  Big Data is already delivering benefits in engineering in particular and the challenges associated with the Internet of Things are not going to result in a reduction in information anytime soon.  Claiming that its all just hype doesn't help move the state of the now forwards and certainly doesn't serve all those use cases which really are Big Data challenges.

But then the roll of doom-mongering sage has never been to be fair and balanced, but instead to take a specific example and declare 'We're all doomed' or 'the end is nigh'.  Where would the book deals be in 'Big Data has many specific use cases but some vendors are using it to hype sales of their technology in places where it doesn't really add value' or to give it a book title 'Salesmen - not always looking out for your best interests'.

Open Source as religion - when the Bazaar becomes a Cathedral

The seminal book on Open Source development "Cathedral and the Bazaar" talks eloquently about the difference between commercial software development and open source development.  In the past few years however there has been another shift, a shift where companies are actively releasing their technology into Open Source as a competitive differentiation.  A claim of 'we are open' because the source code is open.

The selling point then is the number of 'committers' (developers) that the company has on the open source project, this being their selling point because it means they can get your bugs fixed quicker because they have the inside track.

The competition between vendors using exactly the same open source distribution then becomes a question of who is the 'purest' the the vision and who has the most bodies contributing to it.  If an external company takes that source and releases their own version they are not simply frowned upon then are actively prevented from engaging in contribution as this would dilute the corporate messaging of the commercial companies who first established or who mainly contribute today to that open source program.

This isn't an entirely new thing, we used to see it quite a bit with some of the Java pieces and some would argue its related to what Linus does with Linux.  There is however a very big difference.  In those previous cases it was normally a single individual who made the original release and its that individual who then managed that control.  In Linus' case he isn't the commercial arm behind any of these things.

Its natural for this to have happened in the Open Source community as its become a commercial competitive weapon but it does really mean that Open Source is ceasing to become that historical bazaar and is instead in many cases now simply a different cathedral into which rigid company approaches are applied.  Its extremely hard for companies that have locked down millions in VC funding to enable their core market message "we own the code" to be diluted as their Open Source project becomes popular as this would reduce their differentiation and thus their market multiple as they look to IPO.

Open Source remains a strong approach and one that gives companies a level of security if a company ever goes bust, in that the code is still available.  But its quite clear to me that the VC funding that has flooded into the space has really destroyed the previous ad-hoc bazaar approach and instead simply re-created the Cathedral approach but with an Open Source release management system.

MDM isn't about data quality its about collaboration

I'm going to state a sacrilegious position for a moment: the quality of data isn't a primary goal in Master Data Management

Now before the perfectly correct 'Garbage In, Garbage Out' statement let me explain.  Data Quality is certainly something that MDM can help with but its not actually the primary aim of MDM.

MDM is about enabling collaboration, collaboration is about the cross-reference

Why do you do an MDM project?  The answer is to join information between multiple systems and multiple parts of the organisation.  Its so the customer in SFDC is the same as the customer in SAP and in Oracle Financials and when that customer hits the website you know who they are.  Its so the sales person can see all the invoices, orders and other elements related to their customer.  Its so you can see how a product goes through the various parts of the R&D and supply chain processes and track it all the way.

If everything was in one big system with a single database then you wouldn't really need MDM you'd just need data quality to make sure the single record was a good one.  You need MDM because you are attempting to join across systems and business units.  So the real value from MDM is that cross reference that tells you who the customer is and where all the information about them lives in the various systems... even if you never clean any of it.

So this is how you sell MDM to the business, not about data quality which is a secondary benefit, but as something that will enable the business to better collaborate and function more effectively.

Sometimes Quality doesn't count

The reality is that total quality isn't always what the business wants, they know some data is dodgy so the question is how dodgy and knowing that when you use it to make decisions.  Lots of social media is amazingly poor quality, but taken in volume trends can be seen.  What makes it more valuable though is when you can enable that cross-reference between the high-quality and the lower quality so you can see the trends of your customers and products not just trends in noise.

Focus on collaboration, focus on the cross reference, quality will follow
So having said that Data Quality isn't a primary focus it is actually how you enable that pesky cross reference, but you do so only on the information that matters, the core information required for the cross reference.  Thus you get a higher quality core identification of the customer and everyone understands why they are doing it, the quality enables the cross reference which enables the collaboration.

If the business don't care about quality why do you?
Now once you have that quality core, a minimum set of attributes required to uniquely identify the customer, then often you want to expand that quality to more attributes but stop and think.  Have the business asked me? That is quite an important point.  You might think its an absolute disaster that a given attribute isn't used in a standard way, but it could be that no-one in the business gives a stuff, so tell them about the issue but let them decide if they want to spend the money making it better.  If they don't document that they don't so if they come back you can say 'great so lets re-prioritise it' which is much better than 'oh so I spent money doing something that doesn't matter'.

The more you federate the more collaboration matters
The reason that MDM matters is that more and more business is about collaboration, both internal and external, this means that the business value of MDM has really shifted from being about the data quality in reports to being an integral part of how a business works.  Data Quality isn't irrelevant in this world but its turned from being the goal of MDM to being a tool that helps enable the primary goal which is collaboration.  As the need to digitally collaborate with partners and customers increases so the business value of that MDM cross reference increases both in operations and as the bit that helps you link up all of those big data sources to create a global view.

MDM is the Rosetta Stone that enables people to collaborate, so focus on collaboration not quality. 

Lipstick on the iceberg - why the local view matters for IT evolution

There is a massive amount of IT hype that is focused on what people see, its about the agile delivery of interfaces, about reporting, visualisation and interactional models.  If you could weight hype then it is quite clear that 95% of all IT is about this area.  Its why we need development teams working hand-in-hand with the business, its why animations and visualisation are massively important.

But here is the thing.  SAP, IBM and Oracle have huge businesses built around the opposite of that, around large transactional elements, things that sit at the backend and actually do the running of the business.  Is procurement something that needs the fancy UI?  I've written before about why procurement is meant to be hated so no that isn't an area where the hype matters.

What about running a power grid? Controlling an aeroplane?  Traffic management? Sure these things have some level of user interaction and often its important that its slick and effective.  But what percentage of the effort is about the user interface?  Less than 5%.  The statistics out there will show that over 80% of spend is on legacy and even the new spend is mainly on transactional elements.

This is where taking a Business SOA view can help, it starts putting boundaries and value around those legacy areas to help you build new more dynamic solutions.  But here is a bit of the dirty secret.

The business doesn't care that its a mess behind the scenes.... if you make it look pretty

Its a fact that people in IT appear regularly shocked at.  But again this is about the SOA Christmas, the business users care about what they interact with, about their view for their purposes. They don't care if its a mess for IT as long as you can deliver that view.

So in other words the hype has got it right, by putting Lipstick on the Iceberg and by hyping the Lipstick you are able to justify the wrapping and evolution of everything else.  Applying SOA approaches to Data is part of the way to enable that evolution and start delivering the local view.

The business doesn't care about the iceberg... as long as you make it look pretty for them. 

How to select a Hadoop distro - stop thinking about Hadoop

Scoop, Flume, PIG, Zookeeper.  Do these mean anything to you?  If they do then the odds are you are looking at Hadoop.  The thing is that while that was cool a few years ago it really is time to face it that HDFS is a commodity, Map Reduce is interesting but not feasible for most users and the real question is how we turn all that raw data in HDFS into something we can actually use.

That means three things

1. Performant and ANSI compliant SQL matters - if you aren't able to run traditional reporting package then you are making people change for no reason.  If you don't have an alternative then you aren't offering an answer
2. Predictive analytics, statistical, machine learning and whatever else they want - this is the stuff that will actually be new to most people
3. Reacting in real-time - and I mean FAST, not BI fast but ACTUALLY fast

The last one is about how you ingest data and then perform real time analytics which are able to incorporate forecasting information from Hadoop into real-time feedback that can be integrated into source systems.

So Hadoop and HDFS are actually the least important in your future, its critical but not important.  I've seen people spend ages looking at the innards rather than just getting on and actually solving problems.  Do you care what your mobile phone network looks like internally?  Do you care what the wiring back to the power station looks like?  HDFS is that for Data, its the critical substrate, something that needs to be there.  But where you should concentrate your efforts is on how it supports the business use cases above.

How does it support ANSI compliant SQL, how does it support your standard reporting packages.  How will you add new types of analytics, does it support the advanced analytics tools your business already successfully uses?  How does it enable real-time analytics and integration?  

Then of course its about how it works within your enterprise, so how does it work with data management tools, how does its monitoring fit in with your existing tools.  Basically is it a grown-up or a child in the information sand-pit.

Now this means its not really about the Hadoop or HDFS piece itself, its about the ecosystem of technologies into which it needs to integrate.  Otherwise its going to just be another silo of technologies that don't work well with everything else and ultimately doesn't deliver the value you need.