Empowered Holdings has a big impact in the local community, we are active in our local church and non-profit organizations.  We believe in giving 10% or more of our profits to the organizations who perform God’s work with the donations.

In business, we are moving ahead with our growth plans.  We are now working with some vendors to bring Data Vault modeling to market faster and easier than ever before.  If you’ve not yet got your fix for Data Vault, then head on over to http://LearnDataVault.com to purchase the e-book.  You also will receive a downloadable copy of the NDC data Vault National Drug Code), Informatica ETL loading code examples, data models for staging area, and Data Vault along with the data sets to play with.

We are pleased to announce the first of a long-line of on-line e-learning video classes, which will be targeted at the business Intelligence space going forward.  You can find out more about this, by contacting us today.

Cheers,
Dan Linstedt

What is RapidACE?

It is a Rapid Architecture Consolidation Engine. It performs the following:

* Consolidates & Merges multiple disparate data models (source systems, Data Vaults, Staging Areas, Star Schemas)

* Forward engineers NEW data models (from source systems models, it can engineer/build staging areas, Data Vaults, and Star Schemas)

* Tracks and manages cross-reference spreadsheets (from/to, aka: source to target spreadsheets), YES: it reads and writes Excel!

* Generates: ETL Code for Oracle Warehouse Builder and Informatica v9

* Generates: ETL code that generates test data (for Informatica v9)

What are the BENEFITS to you of all of this?

* You save time & money.  You can now produce/generate your data warehouses, data models & ETL code, 90% error free, and in a fraction of the time.

What used to take you 6 months, now takes you 2 to 4 weeks to accomplish, speeding up your data warehousing and business intelligence efforts.

With SaaS model, you’ll be able to sign up and use it on demand with a subscription based model.

Cheers,

Dan L

You’ve really got to believe in God, have faith in your spirituality to get through these hard times.  You can’t look down the end of the road and see all the darkness that’s coming, no – that’s the evil one trying to tear us down, and as God’s children we should stand up, and look skyward.  We should hold fast to our beliefs, we should ban together and use our talents and strengths to shine God’s light in to the world.

So there you have it…    My talents are Data Warehousing, Systems Architecture, ETL, Engineering and Design.  I also had really great writing teachers along the way.  So, I spent the last year and a half or so writing my technical Data Vault Modeling & Methodology book.

If you wouldn’t mind, spreading the word, and telling people where to go – they can sign up for FREE and get a bunch of bonus material at: http://LearnDataVault.com

You can also subscribe to the YouTube channel ID: LearnDataVault

Let me know what you think!

Cheers,
Dan Linstedt

In the Netherlands we had over 240 people attend to find out what’s new, what’s coming, and how they can improve their Data Vault modeling skills.  I was blessed by God to have a productive trip, and believe that God wants me to shine with the IT talents He has given me.  I had a chance to meet many of those folks after the show, and they were all singing the praises of Data Vault.  I owe my success to God, and the blessings He bestowes on me.  He has given me great gifts of architecture, and design – to be used, and shared throughout the world.

The ideas behind the Data Vault methodology and modeling components are:

• Accountability
• Flexibility
• Scalability
• Agility

We want IT and Business to be more accountable, to walk in the light of the facts rather than the light of the “truth” because truth is subjective.  Please note, there is only one truth – and that is God.  I believe that by bringing accountability back to the business, and back to IT we can begin to see corrective actions, wholesome business practices return, and begin to see businesses truly take an interest in their customers.  As customers of these businesses there is nothing worse than to be lied to, or told “marketing fibs”…  The Data Warehouse and BI space really should expose the gaps between the way business is conducted, and the data that is collected.

Without accountability we are nothing.  Without flexibility we cannot grow and change. Without scalability we cannot understand and incorporate new information, and finally, without agility we cannot change fast enough.

These are the principles behind the Data Vault methodology, and the Data Vault Modeling components.  There are over 300 large world-wide enterprises currently engaged in building and deploying Data Vault based projects, I’d encourage you to learn more about it.  You can do so at: http://danLinstedt.com.

In case you are interested, some of the latest customers to build Data Vaults include Sweedish Telecom giant: Tele2 – with over 300 Terabytes of information, 3000 sources to integrate, and a multi-national team from Logica to make it happen.  They succeeded with the Data Vault.  Also included in that list is Vektis, another large company.  And the list goes on: Dutch Railways, Diamler Motors, Department of Defense, US Navy, US Army, ABN-AMRO, SNS Bank, World Bank, Central Bank of Indonesia, and on, and on, and on…

Won’t you join us in the crusade to bring ACCOUNTABILITY and up-standing back to IT in the data warehousing and business intelligence space?

Here’s the abstract of the article:

© Dan Linstedt, 2010 – This document introduces the ideas around real-time information processing in the business intelligence arena.  Insights, rewards and risks are covered at a business level; including costs of acquisition and probabilities of success.  If you’re interested in RT for BI, then this report might be for you.   Source: http://danLinstedt.com – Coaching

Click HERE to download the article: 2010-Information in Real-Time

1) Cloud Computing makes “hardware & software” both: ubiquitous.  You no longer need to worry if it will work seemlessly, or if it works at all – or if the latest version can do this or that with parallel or partitioned computing resources.  Come-on, Grid is out, Grid is old-hat, Grid is for die-hards in their IT shops who “feel the need” to hold on to their own personal server kingdom.

Now, do you still need to worry?  Yes, about your pocket-book.  Which means, as a business user you need to worry about performance, you need to worry about resource hogging processes, you need to worry about efficient use of the resources, you need to put pressure on the cloud providers to go green, to eat less power, all in the name of saving you those razor thin margins.  If you (as a business user) don’t worry about these things, and you jump blindly into the cloud – it can have the effect of eating all your cash reserves.  By the time you figure out what’s happening, you will owe the cloud vendor HUGE sums of money for resources that were used (or better yet: squandered) where they shouldn’t be.

Many are talking about why Cloud Computing is better, faster, cheaper – and yes – many of those things are TRUE, but if you don’t force “software vendors” to clean up their code, be more efficient, use less resources (sound familiar?) then you will have to worry.  Instead of continually pushing on your IT department to “do more with less”, you’ll have to pay attention to software bloat from vendors.  Maybe, they will get rid of features from products that simply sit idle and “eat resources” for the fun of it?

Ok – I’m being a bit unfair, sorry about that.  Let me give you the other side… the Vendor side.

There are a TON of good software vendors out there, and they are all competing for your buck.  They see the cloud coming just like you and I do.  They know that the “better/faster/cheaper” software will earn them more profitability, they know that code-bloat is a bad thing to their bottom line (I hope), they know that if they have “more instances running inside the cloud” that it is better than “one instance that uses all the resources” (because you can equate more instances to more customers, which means more sales for them).

2) Cloud Computing offers a vast array of the “latest” technology, and MUST promise the business community 24x7x365 up-time, or they won’t get the gig…  So, in that case the vendors of the cloud will always be upgrading to newer more green hardware, and you’ll wake up in the morning to notice that your system now runs faster, not slower.

3) Cloud computing offers serious virtualization, which means several instances can be spread all over the globe with high-speed network backbones – so… are you listening cloud vendors?  NO SINGLE POINT OF FAILURE should be a measuring stick by which Cloud Computing service providers are stacked against.  It will be a boon to the back-bone carriers, and it will hopefully lead to new heavy-duty networking devices and more bandwidth (inside the cloud) that you could ever hope for.  You never know if your spreadsheet will be running in Asia today, or Austrailia tomorrow.  Heck, you wouldn’t know where it is running minute to minute.

4) Cloud Computing reaches mobile devices…  Yes, think about it – all that power, single consolidated “systems”, web-interfaces, and light-weight protocols allow you to connect ALL your devices, laptops, and “existing-soon-to-be-outdated” servers.

Is Cloud Computing a panacea?  No, I think not.  There are still plenty of places and plenty of reasons why not to move to the cloud, but that’s for another blog entry. 

I would say for now: to all you executives out there – read about PRIVATE CLOUDS, learn about them, and to those CIO’s feeling the squeeze from Business, this just might be the break you need to make your systems and IT really humm.

I’ll leave you with this thought: IT folks are scared, they are thinking “I’m going to lose my job if the company moves to Cloud Computing”…  They may be right, on the other hand – what they should be REALLY focused on is: “How can I learn the business?  What assistance can I offer the business in moving to cloud computing?  How do I become an expert in the use and evalutation of Cloud Computing Vendors?”  Wake Up People…  If you have Business Knowledge, and can demonstrate value – then they will come to you for advice.  There is still software out there to “choose”, vendors to separate, and functionality “in-the-cloud” to build out.  Not every piece of software has even yet made it to the cloud….  ok – the ones that don’t will be left behind in 2011, but that’s another story too.

Got thoughts on Cloud Computing?  Got a different opinion?  Love to hear, please feel free to comment.

Dan Linstedt
D...@DanLinstedt.com

Basics about these tools and where they came from…

Informatica / PowerCenter / PowerMart

Grew up as a metadata driven, GUI development environment.  The engineers and the founders decided it was a good idea to model ANSI-SQL types of commands in a visually driven drag & drop style.  They also decided that keeping this information in a metadata repository was a good idea.  So they set on their way.  Foundationally strong in Metadata, they also made a key decision to put their metadata in to table structures in standard database systems (hence the repository).  However, they wanted to protect their metadata repository as “intellectual property” that was owned by the company, so they removed indexes, removed primary and foreign keys, and they named some of the elements the same way.

This had it’s upsides and down sides to the tool – especially being metadata driven, the focus was not performance nor parallelism until they reached version 5.  In version 5, they listened to their user base (which was growing by leaps and bounds because of ease of use), and decided to add parallelism, and partitioning, and thus was born the PowerCenter product (they started PowerCenter products in version 3 as an “enterprise class”) – it spawned from the PowerMart line of code.  They went on to mix, match, and re-merge PowerMart back in to one line of product called PowerCenter (which we know and love today).  They added HA, fail-over, clustering, and grid.  Last year they added “PowerCenter” to the cloud computing space. 

Their focus today is still very much metadata (and very strong), but they are now extremely fast and can perform extremely well under the right circumstances.

Abinitio:

  ** NOTE: I’ve never signed an NDA with the company, never had to – the information I retrieved was from reading their publicly filed patents, and talking to their folks (seeing their product demonstrated) at trade shows **

Grew up as a speed mongor.  They did not start with metadata first, their whole purpose in life was bent on making the base OS: a) fault-tolerant b) fast c) partitionable flat file access.   – that’s right, it’s all 100% flat-file driven.  So what they did was create something called the Co-Operating system.  The co-operating system basically is a parasite.  It sits on top of your existing OS, and pluggs in to the I/O interrupt level – it assumes and overrides 100% of all file operations on the machine.  Even files that aren’t “running” through Ab-Initio go through the AB-I interrupts at the file system level.

They then, constructed their own “data description and manipulation language” known as ab-initio scripts (much the same way C-Shell, or K-Shell acts in unix).  They interpret these scripts at the start of a run-request (compile them really), and then figure out mathematically what operations can be split across which resources.  Their only notion of a shared repository (at that time) was the systems-performance metadata.  They gather this metadata from each registered system upon install to make a profile of the machine’s capabilities.  This makes heterogeneous activity possible, and load-balancing mathematically easy.  It’s also why when you upgrade your OS, or upgrade your hardware that you’re asked to re-run the machine profile, and re-register it with the “central brain”.

But as I said, the Co-Operating system is parasitic… It renders the rest of the OS (for all intensive purposes) useless, even though they’d like you to believe it’s seemless and harmless.  Yes, some things still work properly – but others give spurious errors (unknown and untraceable errors).  Anyhow, I digress.  They then developed a GUI that allowed designers to build “data flows” – graphs they call them…  These graphs were then “exported” to the co-operating system (produced if you will).  What that process really does is generate Co-Operating system scripting code.

The unfortunate part of this, is: programmers decided they could take the generated code and make it better/faster… so they tweak it.  Once they tweak it, it’s disconnected from the metadata graph in the GUI… and bang, your maintenance costs rise again – and you’ve got uncontrollable code sprawling across your enterprise.  Now, let’s talk about speed…  BLINDINGLY FAST – IF all your operations are flat-file based.  They do a decent job (like everyone else) of making use of database connectivity, but their flat-file execution rules, it’s a whole different animal.  It’s because of this one trait that Ab-initio can survive and actually compete against Tibco and MQ series, (they are a better fit to messaging systems than they are to ETL batch world).  So long as you like / can manage the code-sprawl.

They built their own metadata database (an internal binary format, they told me once it was a PIC universe… sound familiar?)  but then the source that told me that, soon denied this.  Anyhow, binary or proprietary metadata is a serious problem in the industry.

SSIS…

From Microsoft, really stands for SQLServer Integration Services (formerly known as DTS – Data Transformation Services)…  SSIS has come a LONG LONG way since it’s beginnings, but it too started life as a “tool to call other tools” with, or to execute procedural code in the database.  It was not quite a code-generator, but not quite a metadata driven tool either.  Since then however it’s been implemented in Visual Studio (awesome idea), hooked up with COM objects, and given a full on GUI development environment.  It even has metadata – but where is this metadata?  Alas, the file system.  It’s stored like nearly every other VS project (without team-edition development that is…) in the file system.  An SSIS project is a series of XML documents, albeit seriously complicated and hard to understand (the XML anyhow).  but none-the-less, well formed.

SSIS is not quite a power-house that ab-initio or PowerCenter is, but it’s coming up fast.  Now, if you are a 100% Microsoft shop, SSIS is THE way to go (if you don’t have additional money to spend or can’t justify the cost of another tool).  SSIS is REALLY good at working with SQLServer instances, especially in SQLServer2008 R2.  SSIS is lacking a metadata repository, and SSIS is lacking the ability to hook into a “data model design tool”.  If Microsoft brings these together, it would be the foundation for a great product, especially if their metadata repository is in SQLServer, AND if it is open and documented metadata model.

OWB – Oracle Warehouse Builder

Grew up haphazardly…  One gets the feeling that this started as a pet project somewhere inside Oracle, and grew into the first edition of OWB nearly by accident.  It was almost unusable in the early phases of the product, and even today has serious flaws and quirks that need fixing.  Then along came Sunopsis (who’s exit strategy was clear from the get-go, sell to Oracle).  They sold, and bam – Sunopsis transformations became the backbone of OWB – and some of the Sunopsis GUI got worked in to the nearly 900MB java GUI distributable that eats up client workstations.

OWB is good for one thing (in my opinion): Generating Oracle to Oracle stored procedure code.  Once the code is “deployed” to Oracle as a stored procedure, it’s great – it runs fast, it’s efficient (mostly), and can be monitored.  It has an in-oracle metadata repository which is very positive, but trying to understand where the metadata is stored and how is another question…  All puns aside, I like working with TCL and scripting my mappings, and objects in OWB – but I CANNOT stand their GUI.  It’s like working in your back yard whith your boots stuck in wet concrete – and trying to move a heavy steel pole out of your way.  Again, the up-side is: once you get the steel pole where you want it, it works, it runs, and it runs well.

IF however you are not an ALL ORACLE shop, then perhaps OWB is not for you.  Their flat-file support on the other hand is good, as is their ability to deal with XML.  Now, if you are using Oracle Message Queues – then well, OWB is really good at that.  I like the features that the Database offers – but don’t be fooled, it’s not OWB that offers these features, it IS the database (just like I said previously).

Pentaho ETL

A great product, a great team of engineers, a strong background in metadata.  They too (like Informatica) started with their feet in Metadata.  They focused on it, so much so that they managed to hook together their BI solution to their ETL solution for data lineage.  Ingenious (but they weren’t the first to try this either)…  As of the past 3 years, they (like Informatica) have come a long way to fixing and improving the performance of their engine.   However, nothing beats the speed of a compiled language, and while Pentaho is good, it still can’t do the heavy lifting in the environments I work in (without additional products like Hadoop and cloud computing at the Java level)…

Cloud computing changes everything, and the jury is still out on this one.  They are definately a team to watch with a product to watch… Now only if they had a Data Modeling Tool…. hmmm…

Talend ETL

A great product, another really good team of engineers.  However the way Jasper (the parent company) does business with their open-source is more of a community approach.  Pentaho likes to treat the repository as primary (all tools need to work together), where Talend ETL has it’s own repository – and where applicable will communicate with other tools, but it’s not the primary focus.  Talend ETL (as I understand it) grew up as a high-performance minded engine where metadata comes second; and it shows. 

I’m not saying this is a bad thing, I’m just saying this is where it started.   Note: I see more adoption of Pentaho ETL than I do of Talend ETL – not sure why that is, but I’ve heard from customers that the GUI is hard to work with, and the metadata interconnectivity is hard to deal with.  Maybe there’s something in the learning curve on this one?

Embarcadero ETL

How many of you even knew that Embarcadero had an ETL tool?  DT-STUDIO  Yup… well – don’t know if they still do, but they did for many years.  Embarcadero ETL is a Java based solution (not open-source to my knowledge).   It too grew up as a metadata based solution.  Anyhow, it is connected to the ER-Studio (their data modeling tool).  From that perspective it is very powerful.  Unfortunately due to the fact I’ve not seen it in the major accounts I visit, I haven’t had enough exposure on the tool to comment further (sorry).

DataStage (formerly Formerly Ascential, now IBM)

Ahh DataStage…  The name is what the product does… It “stages” data at each transformation before passing it downstream.  It was built as an ETL GUI (because there was money in the product niche that Informatica was in)…  I am not particularly fond of the “old” DataStage, but the new products that IBM put out continue to get better, so I must applaud IBM for working at this product and improving it tremendously over the years.  DataStage still stages data at each transform, but they’ve gotten wise (it used to do this 100% to disk caches, causing MASSIVE performance problems).  Then, they introduced memory caching – and have improved the performance dramatically. 

DataStage was a good middle-of-the-road product, now it’s engrained in DB2 UDB and Db2 – it’s connectivity to IBM database systems is OUTSTANDING.  It’s ability to run and generate COBOL and Mainframe jobs is unbelievably strong.  It’s work with (other) relational databases leaves quite a bit to be desired.  If you are using mainframe, or mid-range DB2 or AS/400 systems, ADABAS, or other strange products, then DataStage might the product you want.  Of course I think IBM now call it “websphere” – which is a much larger product bundle??   A great resource for this information is my friend Vincent McBurney on IT Toolbox – he’s got a great blog about WebSphere and DataStage.

One of the problems I used to have with DataStage is that every time I wanted to “re-use” a mapping, there was no copy-paste available for the GUI transformation objects.  I think they’ve fixed this by now.  Another issue used to be in deployment of the mappings, from development to test – to get it to a new environment required re-creation by hand.  Then, a 3rd party tool came around to assist with this (still not elegant, but I think it worked).

At the heart of the metadata repository was MetaStage – it used to run only on a PIC UNIVERSE database, a proprietary binary format.  Not sure if it’s still limited to this, or if IBM corrected this by implementing MetaStage and repository on DB2 UDB…  But if it’s still a PIC UNIVERSE, then I would have to complain.  The problems with PIC UNIVERSE are all around.  If the single metadata file get’s corrupted, then ALL your metadata is done for, not just the “one mapping that wen’t bad”.  This is a horrible prospect.  I also never liked the fact that PIC UNIVERSE didn’t have an ANSI-SQL standard driver for it.  Anyhow, those most likely are old problems.

If you have other experience, or if you wish to correct me – please do so by offering comments.  If you wish to discuss other tools that you have used/seen, those comments are welcome as well.  Please feel free to contribute.

Thank-you kindly,
Dan Linstedt
http://danLinstedt.com (home of the Data Vault)
d...@DanLinstedt.com

There are many other Data Vaults that I’m currently engaged in that are well worth their time.  Unfortunately I cannot discuss them at the moment.

What I can discuss is the rise of use of the Data Vault Model in industries that demand compliance and auditability – specifically the banking industry.  Over the past 4 years I’ve engaged with many different banks from around the world, and been able to help them use “triangulation” techniques for gap analysis (between: business perception, source systems & applications, and data warehouses) to reduce costs and improve turn-around time of their IT / BI solutions.

More-over the triangulation allows business users to spot the alignment issues they may have across the three tiers.  I will blog more on the triangulation technique shortly.  It’s used to align points in space, and figure out distances – it is my belief that these mathematical techniques can be used to quantify trouble spots in business.  Once quantified (and the gap has been identified), then and only then a mitigation strategy can be put in place that will be successful.

Do you have any thoughts on this?  Would you like to learn more about these techniques?  Write me a comment, and let me know.

Thank-you,
Dan Linstedt
D...@DanLinstedt.com

Using our unique enablement tool, we can consolidate systems in weeks rather than months.  This includes generating the data movement code to “get the data” out of the original disparate systems, and moving it to the newly consolidated model.

We offer this service for operational systems,  operational data stores, data warehouses, and data marts.

We also track the data lineage through the consolidation effort, and use the lineage to generate “ETL” logic to move the data.  **Note: we are not an ETL tool, we do not run the ETL, we produce ETL code for import to: Informatica, Pentaho, OWB, Data Stage, etc… and we also produce SQL (Stored Procedures) for Sybase, SQLServer, Oracle, and DB2 UDB.

Contact me for more information.
D...@DanLinstedt.com

This may turn you off, and for that: I’m sorry, and stop reading.  This is a story about my personal journey, a testament to where I am today.  I’m not here to force my religious beliefs on you, I’m only here to tell you about my story which has been hidden for far too long.

On the other hand, I have become a strongly spiritual man in my journey along the way.  It’s not been an easy road, and I’m sure there are more challenges yet to come in my life.  But  I want to say “Thank-you” and praise be to God on High for all the blessings He’s bestowed on me throughout my life.  I am writing this just because I think it’s necessary to share my journey on a more personal level with anyone out there who is interested in knowing my story.

1. When I started creating the Data Vault Model…  It was God who gave me the ideas
2. When I started my own consulting company in College…  It was God who told me I can do it.
3. When I coded RapidACE, God gave me the ability to do it.

God is at work in my life, in many many ways.  God has blessed me with two beautiful children and a wonderful wife.  Sure I have trials and tribulations, but God has seen me through all of it.  In high school I was the geek, the nerd, the one who pretty girls ignored…  In College, I was the one who never ‘fit in’, the one who wasn’t invited to house parties.  Sure I had my friends – and some of them were really cool to me.  But I was the one working at electronics companies, writing software, working with computers…

When I went to high school they didn’t have the zero-tolerance policies they have now, I was the one who was picked on….

BUT: You know what?  I turned out great, and know I’m really proud of where I’m at in life and I have God to thank for that.  If you were or are like me AND you are waivering in your faith or life, then know this: GOD through Jesus Can save you.  Miracles DO happen even to normal people.

I can remember one time I was sitting on a plane in NY waiting to take off, we were delayed by a defective part on the plane, about a half hour went by and people were complianing.  A couple beside me reached over and asked: do you think it’s serious?  Do you think we’ll leave soon?  God stirred inside of me to tell them: “5 minutes, we’ll go.” – I prayed about it, and sure enough, 5 minutes later it was all fixed and we were on our way.

But that’s not all God has done in my life.  I owe all my success, every last bit of it to God.  I pray every morning and every night to guide my life – because you see, I am a sinner and only Jesus can save me.

I hope that this has touched some of you out there – you CAN do it, and you CAN achieve success, but you MUST believe, and you MUST keep the faith.