Wednesday, August 24, 2016

Why taking good holidays is good practice

Back when I was a fairly recent graduate I received one of the best pieces of advice I've ever received.  The project was having some delivery pressures and I was seen as crucial to one of the key parts.  As a result my manager was putting pressure on me to cancel my holiday (two weeks of Windsurfing bliss in the Med with friends) with a promise that the company would cover the costs.  I was called into the BIG boss' office with the full expectation that he would put the screws on further and in my head I had a list of additional demands.

"Steve, I've heard that XXX [name redacted] wants you to cancel your holiday"
"Yes, we've got to get the release out and I'm the one who knows most about resolving issues in the team"

This kind of stumped me so I sat there a bit quiet, and here came the best advice ever

"If you cancel this holiday then all anyone will remember is that they can make you cancel holidays, they won't actually appreciate it.  If you go on holiday and it all collapses all everyone will remember is that it collapsed without you, if it goes badly or much harder because you aren't there they will remember that, and if its OK without you then hopefully you have enough talent to not insist that the universe revolves around you.  Go on holiday, don't be contactable and remember that now more senior folks know about you because you didn't cancel your holiday."

Its a mantra I've lived by.  Take your holidays, plan to take them, and take them PROPERLY.  That means

  1. Change your password before leaving and DON'T update it on any mobile devices
  2. Your work laptop does NOT travel on holiday with you
  3. You don't do conference calls, standups, "just 30 minutes a day" or any other nonsense.
Since I've become a manager I think this is more important.  If I can't manage and succession plan to the level that the universe doesn't collapse while I'm away then what sort of manager am I?  I actually look very badly on folks who don't take holidays properly as they risking promoting a "macho hero" culture rather than a decent work/life balance culture.  If you don't recharge your batteries properly, spend proper time with your family then really, what is the point?

Planning for holidays is a sign of good planning, requiring people to cancel holidays is a sign of bad planning.  Requiring yourself to cancel holidays is a sign of extremely bad planning and bad succession management.

If you are so bad a manager that your team can't cope for 2 weeks without you, then you need to look at how you are developing your next level.  If you are so scared that people won't "miss" you when you are out of the office then you really need to check yourself and look at your career ambitions and direction.

Taking a good holiday (vacation for my American colleagues) is one of the key reasons WHY we work, explore the world, meet new people, do new things, RELAX. 

Back to the story at the top, I came back after the holiday to see issues lists and problems, it took 4 days of hard work to get us back to level and everyone saw me as the hero.  Had I cancelled my holiday none of the management team would have known how crucial I was to the team's success, because of the holiday I was rapidly promoted into a new role.  As a developer taking that holiday led directly to people appreciating much more what I did for the team.

These days a key success factor for my team is how they cope brilliantly when I'm not there, I'm not worried that this means I'm irrelevant, it means that as we grow and take on new challenges my leadership team is able to grow and develop and take those challenges on, leaving me free to work on what is next.

Good holidays are good practice.

Monday, August 01, 2016

The ten commandments of IT projects

And lo a new project did start and there was much wailing and gnashing of teeth, for up on the board had been nailed ten commandments that the project must follow and the developers were sore afraid.

  1. Thou shalt put everything in version control, yeah even the meeting minutes, presentations and "requirements documents that aren't even finished yet" for without control everything is chaos
  2. Thou shalt not break the build
  3. Thou shalt never deploy to a shared environment anything that is not under version control
  4. Thou shalt be honest about how long a task will take
  5. Thou shalt test both the happy and unhappy path
  6. Thou shalt automate everything that can be automated and do so early
  7. Thou shalt not re-invent the wheel "just because"
    • Most specifically thou art not Doug Lea, do not create your own threading library
      • If you are Doug Lea then just use your own threading library
  8. Assume Murphy's Law to be always true and never state "that is very unlikely to happen"
  9. Code unto others as you would have them code towards you
  10. Do not screw with the environment configurations without first verifying you are not messing things up for others
The punishment for breaking these commandments is to be relegated to the project management office and be responsible for the many and various Excel spreadsheets report that people outside of the project appear to hold sacred.  


Now because I know that folks will look to interpret these things in many ways lets be clear on a few.

9. Code unto others as you would have them code towards you

This is important, you should write code that is designed to be understood by someone else and written in a way that if you came to it fresh you would be able to understand it.

6. Thou shalt automate everything that can be automated and do so early

This is important, too often people don't automate and more importantly don't automate early, trying to retrofit the automation once the project reaches a certain size.  You should really have everything automated before a single line of code is written, something that 'does nothing successfully'

5. Thou shalt test both the happy and the unhappy path 
8. Assume Murphy's Law is true

Too often I've heard the phrase "that won't happen" followed by "it happened in the first day in production".  This is why its really important to test the unhappy path and to clearly identify the bounds you are going to be operating within.  Hoping that something won't happen isn't a strategy.

Wednesday, January 28, 2015

Making DevOps Business Driven - a service view

I've been doing a bit recently around DevOps and what I've been seeing is that companies that having been scaling DevOps tend to run into a problem: exactly what is a good boundary for a DevOps team? Now I've talked before about how Microservices are just SOA with a new logo, well there is an interesting piece about DevOps as well, its not actually a brand new thing.  Its an evolution and industrialisation of what was leading practice several years ago.

Back in 2007 I gave a presentation on why SOA was a business challenge (full deck at the end) and in there were two pictures that talked about how you needed to change the way you thought about services:

So on the left we've got a view that says that you need to think about a full lifecycle, and on the right you've got a picture that talks about the needs to have an architect, owner and delivery manager (programme manager)
This is what we were doing around SOA projects back in 2007 as a structure and getting the architects and developers (but ESPECIALLY the architects) to be accountable for the full lifecycle.  Its absolutely fantastic to see this becoming normal practice and there are some great lessons out there and technical approaches.

One thing I've not seen however is an answer to what my DevOps team is and how I manage a large number of DevOps teams.  This is where Business Architecture comes in, the point here is that its not enough to just have lots and lots of DevOps teams, you need to align those to the business owners and align them to the structure that is driving them.  You also need to have that structure so one team doesn't just call the 'Buy from Ferrari' internal service without going through procurement first for approval.

So in a DevOps world we are beginning to realize the full-lifecycle view on Business Services, providing a technical approach to automating and managing services that look like the business, evolve like the business and provide the business a structure where they can focus costs where it delivers the most value.

There is much new in the DevOps world, but there is also much we can learn from the Business Architecture space on how to set up DevOps teams to better align to the business and enable DevOps to scale at traditional complex organisations as well as more simple (from a business model perspective) internet companies.

Tuesday, January 20, 2015

Big Data and the importance of Meta-Data

Data isn't really respected in businesses, you can see that because unlike other corporate assets there is rarely a decent corporate catalog that shows what exists and who has it.  In the vast majority of companies there is more effort and automation put into tracking laptops than there is into cataloging and curating information.

Historically we've sort of been able to get away with this because information has resided in disparate systems and even those which join it together, an EDW for instance, have only had a limited number of sources and have viewed the information only in a single way (the final schema).  So basically we've relied on local knowledge of the information to get by.  This really doesn't work in a Big Data world.

The whole point in a Big Data world is having access to everything, being able to combine information from multiple places within a single Business Data Lake so you can allow the business to create their own views.

Quite simply without Meta-Data you are not giving them any sort of map to find the information they need and help them understand the security required.  Meta-Data needs to be a day one consideration on a Big Data program, by the time you've got a few dozen sources imported its going to be a pain going back and adding the information.  This also means the tool used to search the Meta-Data is going to be important.

In a Big Data world Meta-Data is crucial to make the Data Lake business friendly and essential in ensuring the data can be secured.    Lets be clear here HCatalog does matter but its not sufficient, you can do a lot with HCatalog but that is only the start because you've got to look about where information comes from, what its security policy is, where you've distilled that information to.  So its not just about what is in the HDFS repository its about what you've distilled into SQL or Data Science views, its about how the business can access that information not just "you can find it here in HDFS".

This is what Gartner were talking about in the Data Lake Fallacy but as I've written elsewhere, that sort of missed the point that HDFS isn't the only part of a data lake and EDW approaches only solve one set of problems not the broader challenge of Big Data.

Meta-Data tools are out there, and you've probably not really looked at them but here is what you need to test (not a complete list, but these for me are the must have requirements).
  1. Lineage from source - can it automatically link to the loading processes to say where information came from?
  2. Search - Can I search to find the information I want?  Can a non-technical user search?
  3. Multiple destinations - can it support HDFS, SQL and analytical destinations
  4. Lineage to destination - can it link to the distillation process and automatically provide lineage to destination
  5. Business View - can I model the business context of the information (Business Service Architecture style)
  6. My own attributes - can I extend the Meta-data model with my own views on what is required?
The point of modelling in a business context is really important.  Knowing information came from an SAP system is technically interesting, but knowing its Procurement data that is blessed & created by the procurement department (as opposed to being a secondary source) is significantly more valuable.  If you can't present the meta-data in a business structure you aren't going to get the business users able to use it, its just another IT centric tool.

The advantage of Business Service structured meta-data is that it matches up to how you evolve and manage your transactional systems as well.

Thursday, January 15, 2015

Security Big Data - Part 7 - a summary

Over six parts I've gone through a bit of a journey on what Big Data Security is all about.
  1. Securing Big Data is about layers
  2. Use the power of Big Data to secure Big Data
  3. How maths and machine learning helps
  4. Why its how you alert that matters
  5. Why Information Security is part of Information Governance
  6. Classifying Risk and the importance of Meta-Data
The fundamental point here is that encryption and ACLs provide only a basic hygiene factor when it comes to securing Big Data.  The risk and value of information is increasing and by creating Big Data solutions businesses are creating more valuable and therefore more at risk information solutions.  This means that Information Security needs to become a fundamental part of Information Governance and that new ways of securing that information are required.

This is where Big Data comes to its own rescue through the use of large data sets which enable new generations of algorithms to identify and then alert based on the risk and the right way to handle it.  This all requires you to consider Information Security as a core part of the Meta-data that is captured and governed around information.

The time to start thinking, planning and acting on Information Security is now, its not when you become the next Target or when one of your employees becomes your own personal Edward Snowden, its now and its about having a business practice and approach that considers information as a valuable asset and secures it in the same way as other assets in a business are secured. 

Big Data Security is a new generation of challenges, and a new generation of risks, these require a new generation of solutions and a new corporate culture where information security isn't just left to a few people in the IT department.

Tuesday, January 13, 2015

Securing Big Data Part 6 - Classifying risk

So now your Information Governance groups consider Information Security to be important you have to then think about how they should be classifying the risk.  Now there are docs out there on some of these which talk about frameworks.  British Columbia's government has one for instance that talks about High, Medium and Low risk, but for me that really misses the point and over simplifies the problem which ends up complicating implementation and operational decisions.

In a Big Data world its not simply about the risk of an individual piece of information, its about the risk in context.  So the first stage of classification is "what is the risk of this information on its own?" its that sort of classification that the BC Government framework helps you with.  There are some pieces of information (The Australian Tax File Number for instance) where their corporate risk is high just as an individual piece of information.  The Australian TFN has special handling rules and significant fines if handled incorrectly.  This means its well beyond "Personal Identification Information" which many companies consider to be the highest level.  So at this level I'd recommend having Five risk statuses

  1. Special Risk - Specific legislation and fines apply to this piece of information
  2. High - losing this information has corporate reputation and financial risk
  3. Medium - losing this information can impact corporate competitiveness
  4. Low - losing this information has no corporate risk
  5. Public - the information is already public
The point here is that this is about information as a single entity, a personal address, a business registration, etc.  That is only the first stage when considering risk.

The next stage is considering the Direct Aggregation Risk this is about what happens when you combine two pieces of information together, do that change the risk.  The categories remain the same but here we are looking at other elements.  So for instance address information would be low risk or public, but when combined with a person that link becomes higher risk.  When looking at corporate information on sales that might be medium risk, but when that is tied to specific companies or revenue it could become a bigger risk.  Also at this stage you need to look at the policy of allowing information to be combined and you don't want to have a "always no" policy.

So what if someone wants to combine personal information with twitter information to get personal preferences?  Is that allowed?  What is the policy for getting approval for new aggregations, how quickly is risk assessed and is business work allowed to continue while the risk is assessed? When looking at Direct Aggregation you are often looking at where the new value will come from in Big Data so you cannot just prevent that value being created.  So setting up clear boundaries of where approval is required (combining PII information with new sources requires approval for instance) and where you can get approval after the fact (sales data with anything is ok, we'll approve at the next quarterly meeting or modify policy).

The final stage is the most complex its the Indirect Aggregation Risk that is the risk of where two sets of aggregated results are combined and though independently they are not high risk the pulling together of that information constitutes a higher level risk.  The answer to this is actually to simplify the problem and consider aggregations not as just aggregations but as information sources in their own right. 

This brings us to the final challenge in all this classification: Where do you record the risk?

Well this is just meta-data, but that is often the area that companies spend the least amount of time thinking about but when looking at massive amounts of data and particularly disparate data sources and their results then Meta-Data becomes key to big data.  But lets look just at the security side at the moment.

Data Type Direct Risk
Customer Collection Medium
Tax File Number Field Special
Twitter Feed Collection Public

and for Aggregations

Source 1 Source 2 Source 3 Source 4 Aggregation Name Aggregation Risk
Customer Address Invoice Payments Outstanding Consumer Debt High
Customer Twitter Locaiton Customer Locations Medium
Organization Address Invoice Payments Outstanding Company Debt Low

The point here is that you really need to start thinking about how you automate this, what tools you need.  In a Big Data world the heart of security is about being able to classify the risk and having that inform the Big Data anomaly detection so you can inform the right people and drive the risk.

This gives us the next piece of classification that is required which is about understanding who gets informed when there is an information breach.  This is a core part of the Information Governance and classification approach, because its hear that the business needs to say "I'm interested when that specific risk is triggered".  This is another piece of Meta-data and one that then informs the Big Data security algorithms who should be alerted.

If classification isn't part of your Information Governance group, or indeed you don't even have a business centric IG group then you really don't consider either information or its security to be important.

Other Parts in the series
  1. Securing Big Data is about layers
  2. Use the power of Big Data to secure Big Data
  3. How maths and machine learning helps
  4. Why its how you alert that matters
  5. Why Information Security is part of Information Governance